1077-2626 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 1, JANUARY 2020 87

Criteria for Rigor in Visualization Design Study

Miriah Meyer and Jason Dykes

Abstract— We develop a new perspective on research conducted through visualization design study that emphasizes design as a
method of inquiry and the broad range of knowledge-contributions achieved through it as multiple, subjective, and socially constructed.
From this interpretivist position we explore the nature of visualization design study and develop six criteria for rigor. We propose that
rigor is established and judged according to the extent to which visualization design study research and its reporting are INFORMED,
REFLEXIVE, ABUNDANT, PLAUSIBLE, RESONANT, and TRANSPARENT. This perspective and the criteria were constructed through a four-year
engagement with the discourse around rigor and the nature of knowledge in social science, information systems, and design. We
suggest methods from cognate disciplines that can support visualization researchers in meeting these criteria during the planning,
execution, and reporting of design study. Through a series of deliberately provocative questions, we explore implications of this new
perspective for design study research in visualization, concluding that as a discipline, visualization is not yet well positioned to embrace,
nurture, and fully benefit from a rigorous, interpretivist approach to design study. The perspective and criteria we present are intended
to stimulate dialogue and debate around the nature of visualization design study and the broader underpinnings of the discipline.

Index Terms—design study, relativism, interpretivism, knowledge construction, qualitative research, research through design

1 INTRODUCTION

Design study – an approach to applied visualization research [96] – is
now a standard method for conducting visualization inquiry, guided by
validation methods [78, 82], process models [72, 74, 96], scenarios [95],
and an increasing set of representative examples in the literature [9, 43,
56, 62, 77, 79, 84, 107, 117]. In the context of the wider visualization
discipline that is increasingly assessing its practices, the maturing of
design study has exposed a series of provocative, open questions that
we hear researchers asking: What are the research contributions made
through design studies, and do they generalize? What is the value
of specific solutions? If a design study is not reproducible, can it be
rigorous? How do we conduct design study research well, and how do
we assess it? Is design study even research?

Underlying these questions is a strong focus in the community on the
production of visualization software systems within a design study [95].
Process and decision models used by design study researchers prescribe
steps and considerations to design and validate such tools, resulting in a
myriad of validated systems. These open questions, however, highlight
a problem being faced by researchers seeking to use design study to
learn about and express a broader collection of knowledge: process
alone does not provide guidance on important considerations for rigor
and the construction of diverse forms of knowledge acquired through
design [55, 73]. The result over the years has been design studies and
their resulting papers that focus on deployed, working software, rather
than on taking full advantage of the situated, complex, and nuanced
learning that researchers (can) acquire through deep engagement with
people, data, and technology.

In this paper we “separate the criteria from the craft” [110] to support
the broader set of outcomes that can result from design study research.
We propose considerations for achieving rigor in, and constructing
knowledge through, design study that compliment existing processes.
We constructed these considerations from a four-year engagement with
the ongoing and interrelated discourses around knowledge generation
and rigor in social science, information systems, and design. This
debate is deep and extensive, and also contradictory, dynamic, and
imperfect. It is as complex, messy, and nuanced as the richly situated
contexts in which researchers in these fields engage.

Miriah Meyer is with the University of Utah. E-mail: [email protected]
Jason Dykes is with City, University of London, E-mail: [email protected]

We explore the theoretical underpinnings of design study and offer
a new, interpretivist perspective. This perspective emphasizes design
as a method of inquiry into complex, situated, dynamic problems,
and the knowledge achieved through it as multiple in form, various
in range, and inherently subjective and socially constructed. From
this perspective, we propose a preliminary set of six criteria for rigor
in visualization design study that are intended to guide researchers in
constructing, communicating, and assessing rigorous knowledge claims.
We explain why each criteria is an important consideration for rigor,
and identify methods to augment current practices that might help to
achieve them. Our view is that attaining these criteria is challenging
and adopting them may require action by the community, so we pose
several provocative questions that are intended to explore implications,
sharpen views, and stimulate debate about whether and how this new
perspective on design study might be achieved.

But before we provoke, we first attempt to persuade. We develop
the theoretical backdrop of our position in Section 2, summarizing
a range of thinking and debate from the social sciences, information
systems, and design on the nature of knowledge and the ways in which
it is constructed. We position our perspective on design study against
this theoretic backdrop through a series of statements on the nature
of design study in Section 3, and then propose six criteria in Section
4 for establishing rigor from this perspective. In Section 5 we select
three debatable questions that our perspective opens up – there could
be many more – and offer our initial opinions. Finally, we conclude
with a call to the community to critique and debate our work, as well
as the broader philosophical underpinnings of visualization research.

2 THEORETICAL BACKDROP
The perspective on design study that we present in this paper is informed
by a close reading of literature about rigor in social science, information
systems, and design. In this section we provide a brief overview of the
main themes and threads of discourse that informed our thinking.

2.1 Philosophical Positions
The predominant philosophical position in science, computer science,
and visualization is that of positivism, which views reality as singular
and external, on the basis that it can be objectively known. Posi-
tivist research approaches focus on reducing researcher reactivity, and
achieving reliability, replicability, and representativeness [12]. Data
are collected and analyzed with the aim of producing an unambigu-
ous result that is representative of the single reality. Validity criteria
for establishing the truthfulness of results rely on reproducibility and
replication [16], the achievement of which underlies many positivist
approaches. Active discussions on these issues in visualization research
focus on the reproducibility of data, data transformations, interactive

Manuscript received 31 Mar. 2019; accepted 1 Aug. 2019.
Date of publication 16 Aug. 2019; date of current version 20 Oct. 2019.
For information on obtaining reprints of this article, please send e-mail to:
[email protected], and reference the Digital Object Identifier below.
Digital Object Identifier no. 10.1109/TVCG.2019.2934539

Authorized licensed use limited to: Grand Valley State Univ. Downloaded on January 27,2022 at 23:16:15 UTC from IEEE Xplore. Restrictions apply.

88 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 1, JANUARY 2020

exploration, algorithms, and software systems [49, 99], as well as the
reliability of user studies [18, 106].

Discourse in the social sciences and humanities has a long history of
critiquing positivist positions [12, 36, 48], particularly for studies that
seek to understand people and their experiences [66]. These arguments
advocate for a relativist position that considers reality to be multiple
and mind-dependent [102], and the researcher as an active instrument
of the research. In contrast to the positivist position, the subjective
nature of knowledge is a key component and strength of relativist
methodologies [34]. To support a relativist standpoint, interpretivist
approaches view the knowledge that a researcher acquires as socially
constructed – rather than objectively determined – and use methods
such as dialogical approaches that are spoken, written, and interpreted.
Subjectivity is embraced and considered shorthand for the construction
of knowledge through interpretation [30].

Visualization research benefits from both positivist and interpretivist
approaches as it involves multiple types of phenomena and context.
Many perceptual, cognitive, and computational phenomena can be stud-
ied effectively through controlled, empirical studies where objectivity,
repeatability, and prediction are valued and efforts are made to remove
bias and error. Studying people and their considered, complex, contex-
tualized, social reactions to dynamic settings often benefits instead from
relativist approaches that involve subjective interpretation of qualitative
data [30, 57, 61, 63, 107, 108, 113]. A key concern, however, is that
work that is conducted from one position is judged from another [28].
Positivists might question research that involves subjectivity or bias.
Interpretivists, however, are likely to question research rooted in pos-
itivism that does not account for the inherently subjective judgments
involved in most knowledge construction [30]. We argue that the visu-
alization community is missing a broadly shared understanding of how
research emerging from these very different philosophical positions is
undertaken with rigor.

2.2 Interpretivist Criteria
A considerable challenge for interpretivist research approaches is that of
establishing rigor criteria that consider the “creative complexity of the
qualitative methodological landscape” [110]. In their explicit rejection
of the positivist notions of rigor, the seminal work of Lincoln and Guba
[66] established interpretivist criteria for judging the trustworthiness
of research. The criteria credibility, dependability, confirmability, and
transferability are offered as alternatives for scientific validity and
generalizability to instead consider: “How can an inquirer persuade
his or her audiences that the findings of an inquiry are worth paying
attention to?” [66]. The underpinnings of, and methods for achieving
trustworthiness have been considered, debated, expanded, rejected, and
reaffirmed extensively in the literature since [46, 80, 85, 94, 101, 110].

The difficulty in establishing criteria for qualitative, interpretivist
research that is inherently messy, changing, subjective, and context-
specific is in stark contrast to the strong consensus for positivist ap-
proaches that aim for validity, reliability, generalizability, and objec-
tivity – this tension is in part responsible for the undervaluing and
undermining of qualitative work [94, 110]. Resolving this tension has
led to a proliferation of perspectives on rigor criteria in the literature.
For example, Lincoln and Guba subsequently reject much of what was
presented in their original work [46]. Morse calls researchers to reclaim
the validity and generalizability constructs of positivist criteria in the
qualitative realm [80] as have others [85, 101]. And Tracy provocatively
offers eight universal criteria for quality in qualitative studies [110],
which are much used and routinely critiqued [102].

In this paper we take small steps into the conversation through a
proposal of rigor criteria specifically applicable to visualization design
study. Our development of the criteria was informed in-part by the work
and thinking of these social science scholars, and it follows in their
tradition of rejecting wholesale assimilation in favor of more nuanced
criteria suited to a specific approach to research.

2.3 Design Research
The tensions and synergies within and between design and research
are considered in a series of related fields. In the applied field of infor-

mation systems, scholars grapple with the competing needs of design
practice and academic research in the context of developing technology
for and within organizations [97]. Influenced by design science – a
problem-solving paradigm that seeks to analyze, design, implement,
and manage information systems [54, 112] – Sein et al. propose guid-
ing principles through their action design research [97] methodology.
These principles capture a view that the design of information systems
should be both guided by a researcher’s intent and shaped by the orga-
nizational context, which resonates with the goals of design study [96].
The principles are useful for considering the role of people and context
in shaping visualization artifacts within design study, and for recording
and reporting on these effects [72].

The action design research method, and underlying principles, stem
more broadly from action research, an approach that relies on action as
a means for developing knowledge [51, 64]. As a democratic approach,
action research emphasizes researching with people in their everyday,
real-world contexts, not on them through a process that cycles between
planning an intervention, enacting the intervention, observing changes
based on the intervention, and reflecting on the changes in order to
plan for another cycle. Melrose describes the effects of these cycles
on the research process as: “the [researchers and participants] make
mistakes and learn from them, so the research design and questions
are emergent and changeable… [it] is an unrepeatable journey with
unpredictable results and undreamed of conclusions” [75]. Although
action researchers, like other social scientists, question the meaning
of rigor for their work, the literature on action research points back to
Lincoln and Guba’s original notion of trustworthiness [66].

An alternative thread of thinking and discourse on the production
of knowledge through design comes from the research through design
(RtD) community. RtD is an approach embraced by design researchers
and academics who view design artifacts as experiments in future pos-
sibilities and the expression of knowledge a researcher gains about
those possibilities [38, 68, 119]. Importantly, RtD emphasizes the
production of knowledge by means of design activities [103]. The-
oretical work in RtD examines the nature of knowledge generated
through design [6, 21, 59, 69], the ways in which design researchers
design [19, 20, 27, 31], and the relationship of RtD to the goals and
values of HCI [42, 50, 119, 120].

Like relativist positions in social science, RtD strongly rejects posi-
tivist approaches to research. Instead, researchers argue for the need
to embrace a designerly view of knowledge generation that considers
the richness and complexity of the design process, context, and out-
comes [11,41,42,105]. These views place knowledge generation within
“specific, intentional, and non-existing” design contexts [105], and re-
sult in particular, situated outcomes that are subject to a designer’s
unique perspective [42]. Whereas some RtD researchers argue that
methodological standards “threaten to occlude the potency of unique,
embodied artifacts in a cloud of words and diagrams” [41], others ar-
gue for a “philosophical and methodological understanding of what
constitutes the rigor and discipline of design practice in order to better
support practice” [105] (emphasis in original). The synergies between
design research and social science have led to recent calls in the RtD
community for design researchers to more fully and systematically
embrace methodological approaches of the social sciences [39, 87].

2.4 Visualization Design Study

In visualization, design study 1 is defined and described as an applied
methodology by Sedlmair et al. [96]: it is “a project in which visu-
alization researchers analyze a specific real-world problem faced by
domain experts, design a visualization system that supports solving
this problem, validate the design, and reflect about lessons learned in
order to refine visualization design guidelines.” This definition requires
that visualization solutions are designed for a problem that exists in the
world, with domain experts and their data. Through the consideration of
literature, observations, interviews, and their own experiences, design

1Visualization design study is not explicitly related to the academic design

discipline of design study – we note that our reference to design study throughout

this paper is with respect to the visualization community’s definition.

Authorized licensed use limited to: Grand Valley State Univ. Downloaded on January 27,2022 at 23:16:15 UTC from IEEE Xplore. Restrictions apply.

MEYER ET AL.: CRITERIA FOR RIGOR IN VISUALIZATION DESIGN STUDY 89

study researchers build an understanding of a problem domain and the
inherent analysis questions. They operationalize the domain questions
into a data representation and set of tasks [37], which guide the design
of visualization solutions. Design study researchers purposefully assess
their understanding of the domain and the efficacy of their operational-
izations and visualization solutions through checks with collaborators,
data sets, and existing theory and practice. These assessments can be
iterative and multiscale throughout the design process; small, rapid
assessments are embedded in larger, longer-term ones [72]. Reflection
, most often at the end of a study [76], articulates the learning that
occurred to add to the body of visualization knowledge.

Researchers conducting design studies often employ existing visu-
alization models to guide the methodological structure of the study.
Process models such as the nine-stage framework [96], the design-
activity framework [74], or action-design research [72, 97] provide
guidance for the high-level steps a researcher could take to conduct
a design study, with recommendations of specific methods for each
step. Complementing these process models, the nested model [82] is
an often-used design decision model that provides guidance for choos-
ing appropriate approaches for validating a visualization system. This
model categorizes visualization design decisions at four levels, and
identifies validity threats for a designed visualization system at each.

Several forms of knowledge contribution can be achieved through de-
sign studies [96]: a characterization of the problem domain, a validated
visualization design, and improvements to visualization guidelines.
Current guidance emphasizes the importance of reflection in establish-
ing these claims, which, for design study, “is where research emerges
from engineering” [96]. The visualization community, however, has
not reached consensus in the about how to reflect, when to reflect,
or how to improve and judge the quality of the subjective reflective
process [76]. The nested model supports testing the validity of (some)
forms of design study knowledge claims, but it does not provide insight
or guidance into reflectively generating them. Similarly, design study
process models articulate steps to take, but don’t articulate how to pro-
duce meaningful, varied, and valuable knowledge, or what the criteria
might be to judge the resulting knowledge claims.

As a result, there is an emphasis in design studies and their result-
ing papers on the validated visualization design – working software
appreciated by domain experts – rather than on the situated, complex,
subjective, and nuanced learning acquired by visualization researchers
through design study. We argue that methods for developing valid
visualization systems are more accepted, expected, and utilized than
methods for reflecting on the processes to establish knowledge claims.
Thus, in this work we aim to be more explicit about how visualization
researchers can assess their decisions in planning, conducting, and
reporting on design study; what they can learn through design study;
and how others can judge resulting knowledge claims.

3 WICKED SUBJECTIVE DIVERSE DESIGN STUDY
In this section we detail a new, interpretivist perspective on visualization
design study that extends and deepens the existing definition. This
perspective embraces design as a subjective method for constructing
and communicating new knowledge, assuming multiple and mind-
dependent realities. We present this perspective through four statements
on the nature of design study informed by our interpretation, synthesis,
and application of approaches from both the qualitative social sciences
and RtD described in Section 2. For each statement we point to relevant
rigor criteria that we explain and discuss in detail in Section 4.

Design study uses design for inquiry and expression.
Design study researchers learn through the design process. They design
visualization solutions for real-world problems in close collaboration
with domain experts in order to approach the problem in possibly
new ways, and to learn by doing so. Researchers explore possibilities
through broad consideration of design spaces, and express and com-
municate much of their learning through design instances and artifacts,
such as sketches, prototypes, models, and software systems. Therefore,
visualization design study aligns with RtD as “a research approach that
employs methods and processes from design practice as a legitimate

method of inquiry” [120]. In line with RtD, much of what a design
study researcher learns about the molding of materials – combinations
of hardware, software, data, and possibly physical materials – into
the developing solution, and the relationship of this solution with the
problem, is established through the practice of design. This approach
prioritizes finding solutions to a problem through making and proto-
typing over theoretical reasoning [19, 104]. What a designer comes to
know is frequently expressed implicitly through the design itself: its
visual form, its interactive characteristics, and the subtle ways in which
materials are shaped to address the problem [20]. We consider the same
to be true of design study researchers.

Design study researchers are particularly attuned to opportunities for
constructing and testing knowledge through design – ideas, concepts,
encodings, interactions, and their combination – and to engage, observe,
and collect appropriate evidence to explore these possibilities [43, 55,
73]. Taking the perspective of a visualization as a technology probe [60]
offers opportunities to learn about the relationship of people and data
beyond learning about the visualization itself [73]. Ultimately, design
study researchers construct knowledge subjectively through reflective
critical reasoning based upon experience and evidence established
through the study, and against a backdrop of existing knowledge.

Design researchers understand that “the whole point of doing re-
search is to … make knowledge available to others in re-usable
form” [21]. Visualization design study researchers predominately make
their knowledge available through written reports. They aim to produce
explicit and appropriately scoped expressions of knowledge claims that
allow them to be communicated persuasively, effectively, and in ways
that resonate with the community. Reports primarily take the form of an
academic paper including its prose, figures, and other constituent parts.
Additional forms of effective knowledge expression include imagery,
software, digital artefacts and videos with annotations and narrative.

Design as a method of inquiry and expression leads us to suggest
five criteria for rigor – that the design process: is INFORMED by existing
designs to inspire and understand candidate solutions; is ABUNDANT
in observations, designs, and descriptions; produces PLAUSIBLE designs
and interpretations of design processes; generates designs and claims
that are RESONANT; and expresses knowledge claims explicitly through
TRANSPARENT description and evidence.

Design study tackles wicked problems.
Design study researchers design artifacts based on their understanding
and interpretation of a domain problem. A visualization is thus not only
an expression of knowledge, but also a technological representation
of a problem expressed through a potential solution. By developing
visualization designs in close consideration with domain experts and
the context of use – continually reassessing their form, function, and po-
tential [97] – the design study researcher shifts and shapes the solution
to effectively address a problem that is of interest to domain experts.

The iterative, dynamic shaping of the problem and its expression
through the designed solution illustrates the wicked nature [90] of the
problems tackled by design study researchers. Wicked problems are
indeterminant, meaning “there are no definitive conditions or limits
to the design problem” [11]. An important characteristic of wicked
problems is that it is “only in terms of a conjectured solution that
the problem can be contained within manageable bounds.” [19]. The
problem definition is considered a design space just as the solution is,
with progress towards defining one affecting the progress of defining
the other [27, 71, 118]. Wicked problems have unbounded potential for
solutions due to the complexity of design [105], the absence of inherent
stopping criteria [11], and a designer’s articulation of the problem and
solution as one of many possible interpretations [6]. These solutions
cannot be assessed as true or false, but rather as good or bad [11].

Embracing wicked problems as core to design study has several im-
plications. First, wicked problems encourage input from both designers
and domain experts, shaping designs into solutions that are relevant,
meaningful, and interesting. The solutions are inextricably related to
the problem, the design approach, and the people, including the design
study researcher who is defining both problem and solution in ways that
are necessarily interdependent, highly subjective, and fluid. Second, ev-

Authorized licensed use limited to: Grand Valley State Univ. Downloaded on January 27,2022 at 23:16:15 UTC from IEEE Xplore. Restrictions apply.

90 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 1, JANUARY 2020

idence of the changing problem and solution, and their regular shaping
and shifting, is an indication of a strong collaboration between design
study researchers and domain experts working toward a mutually bene-
ficial solution. Instability of a problem definition, identified through
changing focus and expressed through task requirements, is a measure
of success for design study [96]. Third, the design of good solutions
requires the consideration of a broad space of possibilities [96, 105].

The wicked nature of problems that design study tackles requires
two criteria for rigor: An ABUNDANT approach to allow multiple voices
and perspectives to shift and shape design problems and consider a
broad set of solutions; and that evidence of the dynamic process is
reported in TRANSPARENT ways.

Design study is inherently subjective.

What design study researchers learn is personal, subjective, and spe-
cific. The situated and inherently wicked nature of the visualization
design process means that knowledge acquired through design study
can only be understood within the context of its construction. This
context includes not only the views and experiences of domain experts,
datasets, organizational and social constraints, but also a design study
researcher’s own intuition, interests, experiences, and values. The re-
searcher has important effects on the artifacts that she produces, the
problems she addresses, the activities and reactions she observes and
interprets, and the details and knowledge she chooses to report. Visual-
izations, and the visualization design process, are not neutral [17, 26].

Knowledge constructed in this way is inherently interpreted [101,
102] and subject to the many assumptions, values, and commitments
that researchers bring to their work [8]. A relevant position for design
study is that the observable world can never be construed devoid of
and separate from those that observe it [12]. Therefore, we argue for
a relativist perspective to design study, in which observed realities
are accepted as multiple, relative, changing, and mind-dependent [24,
35, 101, 102]. This position contrasts with positivist approaches that
are prevalent in the visualization research community and assume the
researcher to be a distant, objective observer of a singular reality.

Research that takes a relativist standpoint can draw upon established
methods to develop meaningful knowledge from deep engagement in,
and description of, the context in which the observations and expe-
riences take place. These methods utilize subjectivity to support a
researcher in diversifying the perspectives and views she …

1077-2626 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 1, JANUARY 2020 109

Design by Immersion: A Transdisciplinary Approach to
Problem-Driven Visualizations

Kyle Wm. Hall, Adam J. Bradley, Uta Hinrichs, Samuel Huron, Jo Wood
Christopher Collins, and Sheelagh Carpendale

Abstract—While previous work exists on how to conduct and disseminate insights from problem-driven visualization projects and
design studies, the literature does not address how to accomplish these goals in transdisciplinary teams in ways that advance all
disciplines involved. In this paper we introduce and define a new methodological paradigm we call design by immersion, which
provides an alternative perspective on problem-driven visualization work. Design by immersion embeds transdisciplinary experiences
at the center of the visualization process by having visualization researchers participate in the work of the target domain (or domain
experts participate in visualization research). Based on our own combined experiences of working on cross-disciplinary, problem-
driven visualization projects, we present six case studies that expose the opportunities that design by immersion enables, including
(1) exploring new domain-inspired visualization design spaces, (2) enriching domain understanding through personal experiences,
and (3) building strong transdisciplinary relationships. Furthermore, we illustrate how the process of design by immersion opens up
a diverse set of design activities that can be combined in different ways depending on the type of collaboration, project, and goals.
Finally, we discuss the challenges and potential pitfalls of design by immersion.

Index Terms—Visualization, problem-driven, design studies, collaboration, methodology, framework

1 INTRODUCTION
In 1986, McCormick et al. [25] advocated that scientists, engineers
and visualization researchers should form collaborative teams such that
domain needs and processes provide a basis for solving visualization
challenges. However, realizing this vision is complex as there are
multiple paradigms for research involving different disciplines. Kirby
and Meyer [20] characterize multidisciplinary work as addressing chal-
lenges that, while being associated with specific domains, require ex-
pertise from multiple disciplines. In the multidisciplinary paradigm,
“researchers work in parallel with clearly defined roles and specific
tasks that provide added benefit to their disciplinary goal” [20, p.83].
They describe the interdisciplinary research paradigm as addressing
problems lying outside disciplinary confines, requiring the intersection
of multiple disciplines. In this paper, we present a methodology, design
by immersion, that is based on a third paradigm we characterize as trans-
disciplinarity where the lines between visualization researchers and
domain experts blur as individuals move beyond working in a single do-
main. The immersive designer works—partially or fully—in both their
home discipline (visualization or domain) and the ‘other’ discipline (do-
main or visualization). Design by immersion has many benefits. It can
facilitate collaboration and accelerate project development by building
trust and deepening the dialogue between collaborators. From a visu-

I Kyle Wm. Hall is with Temple University, Philadelphia, PA, USA. E-mail:
[email protected]

I Adam J. Bradley is with Ontario Tech University, Oshawa, ON, Canada.
E-mail: [email protected]

I Uta Hinrichs is with the University of St Andrews, Fife, United Kingdom.
E-mail: [email protected]

I Samuel Huron is with the Télécom Paristech, Université Paris-Saclay, Paris,
France. E-mail: [email protected]

I Jo Wood is with City, University of London, London, United Kingdom.
E-mail: [email protected]

I Christopher Collins is with Ontario Tech University, Oshawa, ON, Canada.
E-mail: [email protected]

I Sheelagh Carpendale is with the University of Calgary, Calgary, Alberta,
Canada, and with Simon Fraser University, Burnaby, British Columbia,
Canada. E-mail: [email protected]

alization perspective, it expands the portfolio of existing visualization
design processes in ways that encourage active participation of domain
experts in the visualization process, allowing for the fluid integration of
visualization processes and domains as well as novel perspectives on
visualization. Design by immersion also fosters personal development,
such as the acquisition of new skills and experiences, enabling a better
understanding of different research perspectives and practices. We do
not claim that design by immersion is “better” than existing practises,
but it can offer new ways of looking at visualization design. Design by
immersion is well suited to problem-driven visualization work.

In contrast to technique-driven visualization which aims to create
“new and better techniques without necessarily establishing a strong
connection to a particular documented user need”, the goal of problem-
driven visualization is “to work with real users to solve their real-world
problems” [40, p.2432]. However, problem-driven visualization re-
search comes with challenges introduced by gaps in both knowledge
bases and cultures [18, 26, 27, 32, 33, 40, 41]. In response to these
challenges, the visualization community has developed guidelines for
problem-driven and multidisciplinary visualization projects [40, 43, 45].
Specific design and workshop activities [18, 27], visualization mod-
els [33], and collaborative paradigms [43, 44] have been explored for
engaging with domain experts. However, as Wood et al. note, visual-
ization literature generally creates an opposition between visualization
and domain experts [46]. For example, in their nine-stage design study
methodology framework, Sedlmair et al. advocate that researchers
should clearly identify collaborators’ roles prior to characterizing a
domain and engaging in the design process [40]. Similarly, action de-
sign research [41] (suggested by McCurdy et al. [26] as a visualization
design framework) advocates for clearly assigning roles in collabora-
tive problem-driven design projects. In contrast, a growing number of
visualization case studies report a blurring of the boundaries between
visualization and target domains [1, 11, 12, 16, 46]. In these instances,
the roles of the researchers involved cannot be distinctly classified and
may have even shifted over the course of the collaboration. Even though
Sedlmair et al. [40] call for role definition, they concede that problem-
driven visualization work may involve a single person in the role of
both visualization and domain expert. Similarly, Wong et al. [45, p.1]
note that visualization “tools historically required the users to not only
be domain experts, i.e., have expertise in a specific discipline, but also
have the time and motivation to become visualization experts.”

While previous work hints at the benefits of transdisciplinary ap-
proaches to problem-driven visualization work, methodologies to facil-

Manuscript received 31 Mar. 2019; accepted 1 Aug. 2019.
Date of publication 16 Aug. 2019; date of current version 20 Oct. 2019.
For information on obtaining reprints of this article, please send e-mail to:
[email protected], and reference the Digital Object Identifier below.
Digital Object Identifier no. 10.1109/TVCG.2019.2934790

Authorized licensed use limited to: Grand Valley State Univ. Downloaded on January 27,2022 at 23:06:46 UTC from IEEE Xplore. Restrictions apply.

110 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 1, JANUARY 2020

itate such work are absent in the literature. Using our own experiences
working on distinct transdisciplinary visualization projects, we intro-
duce design by immersion as a paradigm that captures and supports
transdisciplinary approaches to problem-driven visualization where
visualization experts immerse themselves in a target domain in order
to inform visualization processes, and/or domain experts actively en-
gage in visualization design processes to help explore and define visual
solutions to their real-world problems.

Design by immersion is similar to cultural immersion—the direct
experiencing of and engagement with communities, environments,
and/or languages that are different from one’s own. Work in edu-
cation, cultural studies, sociology, and ethnography has shown that
cultural immersion enables a first-hand experience of the target com-
munity/environment/language that remote studies cannot provide [35].
Cultural immersion can lead to a more nuanced understanding of the
target scenario’s characteristics, corresponding processes, and chal-
lenges, and to an increased awareness of one’s own assumptions and
biases in relation to the target domain.

Design by immersion is connected to experiential learning. As
Kolb states: “Knowledge results from the combination of grasping and
transforming experience” [21, p.51] and “[experiential] Learning is the
process whereby knowledge is created through the transformation of ex-
perience.” [21, p.49]. Much has been written about the positive impacts
of experiential learning [8, 15, 22, 23, 30]. Transdisciplinary experiences
offer researchers opportunities to encounter and learn more richly about
other domains and to transfer this knowledge to visualization design.

Leveraging the ideas of cultural immersion and experiential learning,
we consider design by immersion as a new way to describe and guide
collaborative visualization research. In this paper, we illustrate how
design by immersion can be applied to a broad range of collaborative
problem-driven visualization scenarios using a number of case stud-
ies. We discuss the opportunities it supports, including: exploring new
domain-inspired visualization design spaces (e.g., new domain-focused
charts in Case Study #6); enriching domain understanding through per-
sonal experiences (e.g., development of a PhD thesis in Case Study #5);
and building strong transdisciplinary relationships (e.g., ongoing collab-
orative publications in Case Study #4). We provide hands-on guidance
on how to engage in design by immersion with concrete, modular de-
sign activities that can be tailored to different types of projects. Finally,
we reflect on the potential challenges of this method.

This paper contributes: (1) a new methodology to support transdis-
ciplinary problem-driven visualization research, (2) an illustration of
design by immersion using a range of different real-world visualization
case studies with a discussion of the opportunities that design by immer-
sion introduces to visualization research in general, and (3) guidance
for approaching design by immersion in visualization research with a
discussion of potential challenges.

2 RELATED WORK
Our work builds and expands on previous research in problem-driven
visualization and design studies. Design by immersion is related to
grounded evaluation [16] and pre-design empiricism [2], both of which
advocate the use of exploratory qualitative studies to inform design.
However, the activities and themes in our approach go beyond treating
the other domain as the object of study.

2.1 Frameworks to Guide Problem-Driven Visualization
A number of frameworks and guidelines attempt to systematize the
process of problem-driven visualization, conducting design studies,
and working with domain experts. Munzner’s nested model [33] de-
constructs problem-driven visualization design into four components
(domain problem and data characterization; operation and data type
abstraction; visual encoding and interaction design; and algorithm de-
sign) while emphasizing evaluation. Design study methodology was
described in the nine-stage framework by Sedlmair et al. [40], which
provides the high-level stages of a design study. Their work and dis-
cussions highlight the multidisciplinary nature of visualization design
studies. The Design Activity Framework [27] contributes structure

to the process by breaking visualization design down into a set of ac-
tivities (understand, ideate, make, deploy) that consist of motivations,
methods, and outcomes. Wong et al. [45] present a characterization
of domain experts, features of visualization systems for those experts,
and corresponding design guidelines. However, visualization litera-
ture tends to separate visualization and domain experts into explicitly
distinct groups [27, 33, 40, 45]. There is often an implicit or explicit
assumption that the visualization researcher designs a visualization
for a particular domain (or problem) [14, 19, 29, 36]. Such perspec-
tives do not capture the possibility and potential benefits of design by
immersion—a deeply collaborative visualization design process where
domain-inspired solutions arise from transdisciplinary practices and
contribute to all disciplines involved.

Nevertheless, design by immersion aligns with and extends existing
themes in the literature. For example, Simon et al. [43] introduce the
design study Liaison role in which an individual team member (either
a domain or visualization expert) with additional knowledge in the
“other” discipline facilitates visualization design by, in part, serving
as a knowledge conduit between disciplines. There are similarities
between Liaison-supported design studies and design by immersion,
and an immersed researcher is well positioned to serve as a Liaison.
Transitioning to design by immersion involves shifting from a role-
based paradigm with an individual bridging the separate domain and
visualization spheres to a collaborative process that brings together
the two spheres. Design by immersion can be considered as a broader
transdisciplinary approach and mindset that aims at collaboratively
identifying and leveraging synergies between the domain and visual-
ization spheres, allowing visualization design processes and roles to
fluidly evolve, eventually blurring disciplinary boundaries.

2.2 Action Design Research
Considering design research from beyond the field of visualization,
strategies exist that offer a wider view of the role of the target domain
in visualization design. For example, Action Design Research (ADR)
approaches design from the perspective that technological artifacts rep-
resent both design knowledge (visualization theory) and design context
(target domain knowledge and influences from users) [26, 41]. ADR
emphasizes the interconnected nature of: 1) building tools, 2) interven-
ing in the target domain via these tools, and 3) evaluating what has
been built. In ADR, these tasks are tightly bound in successive build-
intervene-evaluate cycles where “evaluation is not a separate stage of
the research process that follows building” [41, p.43]. However, ADR
promotes an artifact-centric perspective to design with a particular fo-
cus on evolution as well as target domain intervention and disruption.
It does not explore transdisciplinary opportunities and their impact.

2.3 Participatory Design
Designers are increasingly focused on including users and stake-
holders in the design process. For example, participatory design
(e.g., [39]) is explicitly multidisciplinary and collaborative. Partici-
patory design has been used by a number of visualization researchers
(e.g., [3, 10, 24, 28, 37]), and Sanders et al. have previously proposed a
framework to organize participatory design tools and processes [38].
Based on Muller’s survey [32] of participatory design approaches, its
characteristic interdisciplinarity (what Muller calls hybridity) stems
from settings, activities and artifacts that encourage the creation of
interdisciplinary spaces where designers and users meet to discuss and
actively work through potentially differing perspectives. In the context
of visualization, participatory design varies substantially from discus-
sions with domain experts [3, 10], through potential users sketching
design solutions [28, 37] and using real domain data as a mediator in
data-driven wireframes and prototypes [24], to domain experts creating
paper prototypes of their ideas [10]. While design by immersion aligns
with ideas of stakeholder involvement as in participatory design, it also
goes beyond them. In design by immersion, collaborations between
stakeholders and designers shift from being structured through inter-
disciplinary spaces, artifacts and decisions to involving the personal
acquisition of skills and field expertise in the visualization domain
and/or the stakeholders’ domain. Design by immersion emphasizes the

Authorized licensed use limited to: Grand Valley State Univ. Downloaded on January 27,2022 at 23:06:46 UTC from IEEE Xplore. Restrictions apply.

HALL ET AL.: DESIGN BY IMMERSION: A TRANSDISCIPLINARY APPROACH TO PROBLEM-DRIVEN VISUALIZATIONS 111

Fig. 1. The immersive skills space. Immersion can result in improving
skills in both visualization and domain areas. Traversal of this space from
novice to expert involves choosing where and how much time to commit
to skill acquisition along the two dimensions.

transdisciplinary transformation of individuals and the opportunities
these transformations present for design. By engaging potential users
and stakeholders in the design process, participatory design invites
crossover between domains, and could serve as a starting point for
transitioning to design by immersion.

2.4 Transdisciplinary Visualization Work
Design by immersion also relates to ideas from the digital humanities
where visualization has started to play an increasingly important role
as a new methodology [17]. Hinrichs et al. have discussed the role
of visualization as a mediator between visualization and humanities
researchers, and as a transdisciplinary speculative process advancing
all disciplines involved [13]. This work emphasizes the importance of
considering visualization as a process that not only enables commu-
nication between different disciplines, but also allows a collaborative
reflection on assumptions inherent in each discipline. Our work builds
on this research by defining and situating design by immersion in the
broader transdisciplinary context of problem-driven visualization.

3 DESIGN BY IMMERSION
This paper is motivated by our—the authors’—experience of trans-
disciplinarity in our own individual collaborative visualization design
projects. We collectively found it difficult to connect these experiences
to the design processes and strategies in the visualization literature.
While working on different projects, engaging with different domains,
all of us had similar experiences with immersion. Reflecting on and
discussing our design approaches, we found that all of us had in some
way engaged with activities typical of the “other” domain and, in this
way, drifted towards becoming members of this “other” community. We
all found this immersion to be challenging but also transformative and
enriching to our projects and our own perspectives on research. While
our approaches and experiences took on different forms, taken together
they illustrate a unique transdiciplinary approach to problem-driven
visualization we define as design by immersion.

Design by Immersion is a methodology for problem-driven visu-
alization design where visualization researchers (or target domain
experts) engage with and participate in the work of another domain
such that visualization design, solutions, and knowledge emerge from
these transdisciplinary experiences and interactions.

This definition is based on our own transdisciplinary visualization
projects as discussed below in Section 4 to illustrate and explicate
design by immersion. We intentionally chose this methodology of
characterizing design by immersion based on a small yet diverse number
of exemplary visualization case studies that we know well as this
enabled us to discuss the nuances of this design approach. While
indications of design by immersion may be present in previous design
studies, extracting traces of this approach from such work would be
difficult without insider knowledge of these studies. We expect that
future instantiating of design by immersion will expand perspectives
on and understanding of design by immersion.

From a visualization perspective, design by immersion means to
immerse oneself in the target domain and to engage with the data
and analysis processes in the ways that domain experts do, to inform
visualization processes and design. From the perspective of a domain
expert, it means to engage with visualization as a design and thinking
process in order to help explore and define approaches or solutions
to a problem. This method is a flexible approach to problem-driven
visualization design that can take on many forms, depending on the
disciplines and types of collaborators.

Our definition exhibits four themes, and we use them as lenses to
connect our case studies to our design by immersion definition.

Communal. Researchers enter into each others’ domains, and
existing communities with their own actors and cultures.
Personal. Researchers become intimately concerned with, affected
by and personally involved in the other domain.
Active. Researchers actively engage in the other domain,
participating in domain activities, not just observing activities.
Emergent. The processes and results of this approach have their
origins in and emerge from transdisciplinary interactions between
visualization and the target domain.
A key characteristic of design by immersion is that researchers trans-

form and enrich their knowledge and skills through transdisciplinary
experiences as demonstrated in Figure 1. Each discipline is represented
as an axis. The expertise of collaborators in their own “home” discipline
(visualization or target domain) can vary, as represented in these axes
that span from “novice” to “expert”. There is variation among visual-
ization researchers (and domain experts) in terms of expertise, such as
graduate students vs. senior researchers. It is common for visualization
experts to have minimal knowledge or expertise in the target domain,
and vice versa. These boundary cases are represented by the red and
blue rectangles (see Figure 1). Of course, there are people who are
novices in both the target domain and visualization, corresponding to
the lower bound on the knowledge space (the orange square in Figure 1).
Design by immersion results in increasing one’s skills and knowledge
in disciplines other than one’s own. Maximum time commitment can
even result in becoming a dual citizen (the purple square in Figure 1),
although this is a rare achievement and not a required result of design
by immersion. Design by immersion typically leads to a drift of (some
or all) involved researchers within this knowledge space (as discussed
in [11]). We do not suggest that there is a correct trajectory to take, this
depends on the project and the people involved. However, an awareness
of this knowledge space and where one would locate oneself can help
reflection on collaborative practices and design activities already taking
place. We will identify how we have “drifted” through this knowledge
space as we discuss our case studies.

The case study descriptions that follow culminate in a series of
descriptive tables which can be used as a starting point for new projects.
Table 1 lists activities that can benefit a research project. Table 2
provides guidance for understanding the stages of research for those
activities. Table 3 describes possible themes and reflective questions to
consider, and Section 7 describes potential problems to watch out for.

4 IMMERSION CASE STUDIES

Our case studies cover a wide variety of domains, including compu-
tational linguistics, medicine, literary analysis, transport studies and
chemistry. While the literature lacks a characterization of design by
immersion, we draw on a number of existing studies that exemplify
immersion [1, 2, 12, 16, 46]. This is not intended as an exhaustive
characterization of immersion but rather to illustrate how the ideas in
this paper arose from independent places and projects, while capturing
the richness and multifacted nature of design by immersion. We use
each case study as a way to reveal transdisciplinary activities using
Letter-# to refer to the activities in Table 1. These activities have been
grouped as data analysis, study methods, prototyping, learning about
the other domain, and communicating across domains, though some
could be classified in multiple categories. Some activities appeared
across multiple case studies as highlighted in Table 1, so we focus on
detailing a subset associated with each case study.

Authorized licensed use limited to: Grand Valley State Univ. Downloaded on January 27,2022 at 23:06:46 UTC from IEEE Xplore. Restrictions apply.

112 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 1, JANUARY 2020

Table 1. Activities and their connections to the opportunities of design by immersion. Activities are classified as Data analysis, Study methods,
Prototyping, Learning about the domain, and Communicating with collaborators. Numbers in the three columns are linked to the case studies.

Design by Immersion Activities Enrich DomainUnderstanding
Explore

New Spaces
Build

Relationships
D-1 Undertake domain-specific data analysis independently 1,4,5,6 5

D-2 Enrich datasets meaningfully by deriving new data 5 4,5 4

D-3 Analyze data collaboratively with domain experts 3,4,5,6 3,4,5,6

S-1 Observe domain experts practices unobtrusively in situ 2,6

S-2 Keep documentation of experiences 1,6 4

S-3 Interview collaborators 2

S-4 Attend meetings and discussions in the other domain 2,5 5 5

P-1 Develop visualizations in the context of evolving collaborative research projectswith multiple disciplines 5 4,5 4,5

P-2 Develop visual encodings that explicitly incorporate and take inspiration fromdomain knowledge and practices 3 4,6

P-3 Start ideating early in the design process using pre-existing domain knowledge 5 1,3,4,5

P-4 Iterate rapidly and collaboratively on designs by leveraging informal domainexpert feedback 5 1,4,5

P-5 Self-critique designs from the visualization and domain perspectives 3,4,5,6

L-1 Engage directly with both domain-specific and visualization literature 3,4,5,6 3,5 3,5,6

L-2 Gain broader exposure to domain concepts beyond domain problem 5,6 5,6 5

L-3 Establish domain-based design considerations for visualizations 5 5,4,6 5

L-4 Receive informal training from collaborators 1 5

L-5 Participate in simulations of domain work 2 2

C-1 Use language that resonates with collaborators 3,5 4 4,5,6

C-2 Relate across disciplines through common knowledge and experiences 5 1,4,5,6

C-3 Engage in informal peer-to-peer communication with domain experts aboutdomain science and visualizations 4,5,6 4,5,6 4,5,6

C-4 Translate concepts and material for design team members coming frompredominantly visualization or target domain backgrounds 3 3,6

C-5 Brainstorm with collaborators about methods that would best elicit implicitknowledge 2 2 2

4.1 Case Study #1: Apprenticeship

One approach to design by immersion is for a visualization researcher
to participate in domain activities (an upward movement in Figure 1).

Context: Collins et al. [6] recount a visualization researcher us-
ing immersive observation to become an apprentice in the context of
computational linguistics, and thus gain first-hand experience in sta-
tistical machine translation. In this case, the visualization researcher’s
first-hand experiences in the target domain provided a contextualized
understanding of it, which informed the visualization design process.
The researcher’s experiences were a combination of being an immersed
intern in the team and consciously exploring the current use of visual-
ization like a type of qualitative pre-design study.

Activities and Design Discussion: In this case, the lead visualiza-
tion researcher had some prior knowledge of computational linguistics.
The domain expert collaborators worked specifically on statistical ma-
chine translation and trained the visualization researcher to carry out
common domain analysis tasks (L-4). The visualization researcher
subsequently engaged in independent analysis with standard tools,
generated his own domain-specific findings (D-1, Personal, Active),
and validated his findings with domain experts. By using pre-existing
domain knowledge, the immersive researcher was able to start the vi-
sualization ideation process early, sharing visualization sketches with
computational linguistics experts on a daily basis (P-3, Communal).
Through shared work environments and meetings, this period involved

rapid, collaborative design iterations tightly coupled to informal domain
expert feedback (P-4), quickly converging on the design of Bubble Sets
(Emergent) [6]. During these experiences, the immersed researcher
kept a journal, which the research team subsequently leveraged to
understand the domain’s data and work practices (S-2).

Benefits and Impact: Insights into the domain problem went be-
yond those elicited through initial interviews with domain experts.
Through experiencing the analysis process first-hand, the visualiza-
tion researcher was better equipped to identify opportunities where
interaction design and visualization could improve the …

98 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 1, JANUARY 2020

Data by Proxy — Material Traces as Autographic Visualizations
Dietmar Offenhuber Northeastern University

Phenomenon

Data

Analysis

gap

start
here

pattern
exploration

interpretation

Phenomenon

Data

Analysis

gap

start
here

revealing
traces

interpretation Phenomenon

Data

Analysis

revealing
traces

interpretation

pattern
exploration

interpretation

Information Visualization Autographic Visualization Combined Model

Fig. 1. Traditional information visualization starts with the exploration of a data set to find inherent patterns. The represented
phenomenon is interpreted based on what is found in the data, but the connection between data and phenomenon remains hidden.
Autographic visualization starts with the phenomenon and explores the data generation process through material traces. A gap remains
between the interpretation of traces and more complex forms of computational analysis. A combined model uses autographic principles
of revealing traces to re-contextualize data with the phenomena they supposedly describe.

Abstract—Information visualization limits itself, per definition, to the domain of symbolic information. This paper discusses arguments
why the field should also consider forms of data that are not symbolically encoded, including physical traces and material indicators.
Continuing a provocation presented by Pat Hanrahan in his 2004 IEEE Vis capstone address, this paper compares physical traces
to visualizations and describes the techniques and visual practices for producing, revealing, and interpreting them. By contrasting
information visualization with a speculative counter model of autographic visualization, this paper examines the design principles for
material data. Autographic visualization addresses limitations of information visualization, such as the inability to directly reflect the
material circumstances of data generation. The comparison between the two models allows probing the epistemic assumptions behind
information visualization and uncovers linkages with the rich history of scientific visualization and trace reading. The paper begins
by discussing the gap between data visualizations and their corresponding phenomena and proceeds by investigating how material
visualizations can bridge this gap. It contextualizes autographic visualization with paradigms such as data physicalization and indexical
visualization and grounds it in the broader theoretical literature of semiotics, science and technology studies (STS), and the history
of scientific representation. The main section of the paper proposes a foundational design vocabulary for autographic visualization
and offers examples of how citizen scientists already use autographic principles in their displays, which seem to violate the canonical
principles of information visualization but succeed at fulfilling other rhetorical purposes in evidence construction. The paper concludes
with a discussion of the limitations of autographic visualization, a roadmap for the empirical investigation of trace perception, and
thoughts about how information visualization and autographic visualization techniques can contribute to each other.

Index Terms—Traces, indexicality, data physicalization, proxy data sources, data materiality

1 LIMITATIONS OF SYMBOLIC REPRESENTATION

Information visualization is, by definition, bound to the domain of
symbolic representation—information encoded in numbers and letters.
Symbolic abstraction offers many advantages, including generalizable
visual methods for pattern discovery and visual computation. How-
ever, visualization can only begin when data already exist. As a result,
the material processes and circumstances of data collection remain
largely hidden. Without additional context, a falsified data set may be
indistinguishable from an authentic one. InfoVis proceeds to further ab-
stract an already reductive set of observations and investigates external
phenomena by looking inwards: seeking patterns and inconsistencies
in the data sets representing these phenomena. As a result, data and
phenomenon are separated by a gap that obscures their relationship and
shared history (Fig. 1, left).

G ���*#�( ����$�+��( �) -�*�
%(*���)*�($ �$�,�()�*/; �@#��”<
�;%���$�+��(D$%(*���)*�($;��+;

There are, however, situations where the circumstances of data gen-
eration are of central interest and subject to public controversy. In the
case of climate change and environmental pollution, for example, it is
not just the discovery of hidden patterns that matter, but the question
of how even the most obvious patterns relate to the physical world.
Since both climate and pollution are statistical concepts based on long-
term averages and threshold values, data visualization seems to be the
obvious mode of representation [17, 61]. Yet, the mantra of “above
all else show the data” [69] is complicated by the fact that data gain
meaning only through context and that data and context cannot always
be distinguished. The visual languages of information visualization,
however, tend to further decontextualize data for the sake of pattern
discovery. Climate change skeptics often circulate charts based on data
from ice cores and other paleoclimatic proxies to support arguments
for “global cooling,” yet showing data accurately and providing the
correct references. They mislead by foregrounding patterns that cannot
be correctly interpreted without a deeper understanding of paleoclima-
tology and its conventions. The obvious response to this issue is that
complex matters require more complex explanations and information
displays. After all, dealing with complexity is considered one of the
main strengths of information visualization. But this strength is of little
help if it is the relationship between the data set and physical reality

1077-2626 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Manuscript received 31 Mar. 2019; accepted 1 Aug. 2019.
Date of publication 16 Aug. 2019; date of current version 20 Oct. 2019.
For information on obtaining reprints of this article, please send e-mail to:
[email protected], and reference the Digital Object Identifier below.
Digital Object Identifier no. 10.1109/TVCG.2019.2934788

Authorized licensed use limited to: Grand Valley State Univ. Downloaded on January 26,2022 at 23:08:51 UTC from IEEE Xplore. Restrictions apply.

OFFENHUBER: DATA BY PROXY — MATERIAL TRACES AS AUTOGRAPHIC VISUALIZATIONS 99

that is called into question.
To address the material circumstances of data collection, it is helpful

to consider a broader definition of a datum. Paleoclimatologists use a
vast array of data proxies, including tree rings, ice cores, and bioindica-
tors such as plants, animals, and microbes [35, 44]. These proxies are
not just the raw material for creating data sets; we can also see them as
physical forms of data. The narrow focus on symbolically encoded data
in computer science is not universally shared across fields. Archaeolo-
gists describe the artifacts they extract from a field site as data, and also
physicists frequently treat information as a material property. Philoso-
pher Luciano Floridi defines a datum as a “lack of uniformity,” echoing
cyberneticist Gregory Bateson who defined a bit of information as the
“difference that makes a difference” [3, 19]. The lack of uniformity
includes any kinds of contrasts that manifest themselves long before
any act of encoding.

Ice cores and tree rings are not just material, but also visual forms of
information that can be investigated through visual methods (Fig. 2).
As conspicuous records of slow processes, their layers bear testament
of past conditions. In this paper, we regard them as autographic visu-
alizations: phenomena that reveal themselves as visible traces. Are
all traces—and therefore almost anything in the world—autographic
visualizations? A focus on practice avoids such a semantic dilution.
Autographic visualization is a technique more than a thing: the appli-
cation of specific methods to reveal environmental information as a
trace, even if it involves only a particular observation skill. To observe
tree rings, one has to cut into a tree and prepare the sample in specific
ways. Beyond self-evident visual patterns, also non-visual qualities can
be visualized, for example, by adding a tracer substance. Autographic
visualization compares such material transformations to the process of
visualizing data.

Fig. 2. GISP2 ice core section showing annual layer structure (cropped),
illuminated from below. Source: ftp://ftp.ncdc.noaa.gov/pub/data/
paleo/slidesets/icecore-polar/

2 AUTOGRAPHIC VISUALIZATION
We present autographic visualization as a counter-model to data visu-
alization, focusing on material rather than data encoded in numbers
and symbols. It is a counter-model not in the sense that it excludes or
replaces data visualization—there is considerable overlap between the
two models—but in the sense that it clarifies the characteristics of each
model through this juxtaposition.

We refer to autographic visualization as a set of techniques for re-
vealing material phenomena as visible traces and guiding their interpre-
tation. Designing an autographic display means setting the conditions
that allow a trace to emerge. We understand a trace as any transient or
persistent configuration of matter presenting itself to the senses.

A central goal of autographic visualization is to make environmen-
tal information legible and the processes of data collection and their
underlying causalities experiential and accountable. Since a data set
is the outcome rather than the starting point (Fig. 1, middle), auto-
graphic visualization cannot rely on the representation of data. It is
non-representational: rather than re-presenting something absent, the
phenomenon presents itself. Autographic visualizations can be acci-
dental, such as the desire paths across grass areas in the city or the
uneven traces of wear on a staircase or a computer keyboard. But in

Table 1. Comparison between InfoVis and Autographic Visualization

InfoVis Autographic Visualization

Role of
symbolic data

Begins with data Ends with data

Focus Inwards: reveals
patterns within data

Outwards: reveals the
process of data generation

Role of
representation

Representational: visual
marks stand for a
phenomenon

Non-representational: the
phenomenon presents
itself

Role of design Mapping data to visual
variables & layouts

Elucidating qualities of a
phenomenon

general, they are the outcome of design operations that aim to reveal,
isolate, amplify, conserve, and present material traces as records of past
processes and events. For example, the display of the sundial [54] is
a product of a natural phenomenon untouched by human intervention.
At the same time, it is a computational device designed to calculate not
only the time of day but also month and season. Its display often incor-
porates a calendar—a classic form of data visualization—geometrically
aligned with the path of the sun in the particular location.

Autographic visualization techniques draw from a long history of
epistemic and material cultures that deal with the visual interpretation
of traces, symptoms, or signatures as forms of material evidence. Its
practices range from scientific experimentalism to ancient techniques of
hunting, navigating, and healing. This paper is based on two premises.
First, the diverse space of practices engaging with traces can be gener-
alized into several distinct design operations. And second, these visual
operations of autographic visualization are closely related to the modes
of exploration facilitated by information visualization.

While the interpretation of medical symptoms, the design of experi-
mental systems, or the design of shape-changing materials are usually
considered in isolation, autographic visualization identifies common
visual strategies across all of these practices. Table 1 summarizes the
main differences between InfoVis and autographic visualization. To
avoid confusion, we use symbolic data to refer to digital data.

Despite these differences, there is a close kinship between au-
tographic and information visualization—both are rooted in the
same visual culture and take advantage of similar perceptual mech-
anisms [25, 47]. Foundational literature in visualization and HCI fre-
quently invokes natural phenomena as metaphor or inspiration. Whether
charts and graphs should be viewed as abstractions of natural phenom-
ena based on shared organizational principles or as metaphorical refer-
ences will not be elaborated here. However, it is worth noting that both
InfoVis and autographic visualization were at one point considered to
be the same approach. Etienne-Jules Marey’s late 19th century Methode
Graphique encompasses both the charting of statistical information and
the construction of self-registering devices for recording blood pressure,
the flight of birds, or the turbulence of air. In pursuit of his declared
goal to capture “the language of the phenomena themselves,” Marey’s
pioneering work included autographic devices such as the wind tun-
nel and, most prominently, his invention of chronophotography [43],
inspiring other non-mimetic uses of photography [18].

Analog information visualizations are often at the same time phys-
ical traces. Mechanically excited by seismic movements, a simple
seismometer produces a classic line chart. This dual role creates a
conceptual ambiguity that blurs the boundary between InfoVis and au-
tographic visualization. If the line chart produced by the seismometer is
a physical trace, what about a satellite image, what about the electrical
charge generated by a digital sensor connected to a computer? The
difference between an analog and a digital medium is not relevant for
the underlying causality since both devices operate in a deterministic
way. From this perspective, many symbolic data sets indeed share
the character of a material trace; the material aspects of data collec-
tion inscribe themselves, sometimes unintentionally, into the data data
set [16].

This can be illustrated through a public data set of GPS traces of drop-

Authorized licensed use limited to: Grand Valley State Univ. Downloaded on January 26,2022 at 23:08:51 UTC from IEEE Xplore. Restrictions apply.

100 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 1, JANUARY 2020

off and pick-up locations of NYC taxis. Plotting the data set in Cartesian
space yields, unsurprisingly, a figure that resembles a map of the city.
Some areas on this map, however, appear blurrier than others: an artifact
of diminished GPS reception between tall buildings. In other words, the
two-dimensional geographic datum contains hidden information about
the three-dimensional shape of the city. But this latent information is
only accessible if the materiality of GPS is understood and considered.
A material reading that takes advantage of such artifacts, or “dust” in
the data [40], differs from a classic approach of cleaning the data set
by excluding obvious errors, e.g., points that fall into the ocean or
within buildings. While information- and autographic visualization
may differ in the length of the causal chains that link phenomenon and
representation, the autographic perspective can to some extent also be
applied to digital information, further explored in Section 7.2.

3 THEORETICAL PERSPECTIVES ON MATERIAL TRACES
Classic semiology, in many ways foundational for information design
and visualization [6, 10, 60], offers a framework for analyzing physi-
cal traces. Philosopher Charles Sanders Peirce differentiates symbol,
icon, and index as three kinds of (non-exclusive) relationships a sign
can have with a corresponding object or concept in the world. The
symbol is linked to its object based on arbitrary convention; the icon
is based on a relationship of resemblance; and in the case of the index,
the relationship is an existential connection such as the causal link
between a footprint and the person that left it [52]. While icons (Peirce
includes diagrams in this category) and symbols play a prominent role
in information design and visualization, indexical signs appear only
implicitly; for example, as patterns and signals in data sets.

Indexical phenomena have been explored in HCI, ubiquitous com-
puting, and to a lesser extent, information visualization [45, 48, 51, 62].
The application of indexicality to physical traces, however, is some-
what limited by the central role of linguistic concepts in semiotic theory.
Peirce, for example, describes a pointing finger, a physical trace, and
the word “there” as equivalent examples of indexical signs. By relying
on the sign as the universal vehicle of meaning, semiotic perspectives
reduce the trace to its role as a signifier. Scholars have critiqued the
semiotic model of representation, in which meaning is conveyed by
signs that stand for concepts in the world. To paraphrase historian Lor-
raine Daston, the proposition of a one-to-one correspondence between
a sign and its object turned out to be as useless as the Borgesian 1-to-1
map that fully covers the territory [12].

When considering all the processes, actions, and material conditions
involved in exploring traces, it is not always useful to make explicit
what exactly constitutes a sign and how it is used to generate meaning.
Scholars in science, technology, and society (STS) have formulated
alternative perspectives that focus on the performative and embodied
modes of cognition with regard to the roles of traces and trace-making
in the history of science. Bruno Latour describes data, traces, and
visualizations as immutable mobiles: aspects of the world that have
been stabilized, flattened, and made mobile to support arguments in
scientific discourse [36, 37]. In a similar vein, Hans-Jörg Rheinberger
speaks about epistemic things: objects manipulated in the laboratory
that should not just be regarded as samples collected from the world,
but as materializations of research questions and scientific models that
are embodied in the countless transformations applied to them [56].

The notion of the trace as objective evidence and science as a process
of trace-making has blossomed in the 19th century paradigm of mechan-
ical objectivity. Charting the history of objectivity through scientific
atlases and visualization, historians Daston and Galison describe the
paradigm as a pursuit to develop modes of inscription that create pure
and objective visualizations without human intervention, even if just
for the sake of removing dirt and imperfections [13]. Culminating in
the work of E. J. Marey, mechanical objectivity still resonates in con-
temporary efforts to develop canonical visualization principles based
on scientific criteria.

Mechanical objectivity in its purest ambition of tracing “nature’s
pencil,” however, was bound to fail due to the indispensability of narra-
tive explanation and the ambiguous nature of the trace. Historian Carlo
Ginzburg describes the interpretation of traces, clues, and symptoms

as a method of conjecture rather than computation [22]. Philosopher
Sybille Krämer locates traces “at the seam of where the meaningless
becomes meaningful,” embodying meaning through material configu-
ration rather than verbal attribution. In her understanding, traces are
not found, but constructed in the act of reading: a trace is whatever
is recognized as a trace [33]. Contemporary thinkers under the um-
brella of new materialism, however, do not insist on the centrality of
the human observer [71]. Distinct from both realist (focusing on the
external world) and anti-realist (focusing on the relationships among
signs) perspectives, Karen Barad’s concept of agential realism con-
siders the human subject as a part, but not the center of an external
phenomenon [2]. Avoiding any dualism between objects in the world
and their representations, Barad understands a phenomenon as an on-
going process of what she describes as intra-actions rather than a fixed
set of objects and their relationships.

Translated to the subject at hand, this implies that autographic visu-
alizations are not stable artifacts whose correct interpretation is just a
matter of visual literacy, but phenomena that emerge from a recipients’
extensive engagement with the world and with the knowledge of others,
like a hunter who learns to spot latent animal tracks that are not just
invisible but non-existent for an unskilled person. Philosopher Michael
Polanyi aptly describes how a complex trace can depend on theoretical
concepts and language [55]:

Think of a medical student attending a course in the X-
ray diagnosis of pulmonary diseases. He watches, in a
darkened room, shadowy traces on a fluorescent screen
placed against a patient’s chest, and hears the radiologist
commenting to his assistants, in technical language, on the
significant features of these shadows. At first, the student is
completely puzzled. [. . . ] The experts seem to be romancing
about figments of their imagination- he can see nothing
that they are talking about. Then, as he goes on listening
for a few weeks, looking carefully at ever-new pictures of
different cases, a tentative understanding will dawn on him;
he will gradually forget about the ribs and begin to see the
lungs. And eventually, if he perseveres intelligently, a rich
panorama of significant details will be revealed to him (p.
106)

4 AUTOGRAPHIC VISUALIZATION NEIGHBORS
Autographic visualization shares a space with other visualization mod-
els concerned with physical information displays, embedded in physical
environments and contexts of action [78]. They can be seen in the tra-
dition of ubiquitous computing and its explorations of tangible media,
ambient and situated displays [27, 72, 73, 80].

Within the information visualization discourse, the field of data
physicalization is closest to the concept of autographic visualization.
Data physicalization investigates three-dimensional physical embodi-
ments of information and their possible advantages for the communi-
cation and exploration of data [30]. Unlike autographic visualization,
however, physicalization (or physical visualization) is a data-first ap-
proach. As Jansen et al. explain: “Traditional visualizations map data
to pixels or ink, whereas physical visualizations map data to physi-
cal form” [29]. Data physicalization aims to take advantage of the
cognitive processes involved in examining, manipulating, and con-
structing three-dimensional objects that may not be accessible through
visual observation of two-dimensional representations. The goal of data
physicalization is therefore epistemological—supporting data analysis—
while autographic visualization emphasizes ontological questions such
as what constitutes a datum and how it relates to the world.

Based on the Peircean concept of the index, indexical visualization
presents a design space spanned by the dimensions of symbolic and
causal distance [51]; the former describes the amount of symbolic me-
diation used to transform a phenomenon into a display, the latter the
number of transformations in the causal chain. Despite its short causal
distance, a simple seismometer involves a high degree of symbolic
mediation; its line chart can no longer be connected to the phenomenon
without knowledge of the process that created it. Conversely, an ambi-
ent display that mimics the outdoor sky based on weather data would

Authorized licensed use limited to: Grand Valley State Univ. Downloaded on January 26,2022 at 23:08:51 UTC from IEEE Xplore. Restrictions apply.

OFFENHUBER: DATA BY PROXY — MATERIAL TRACES AS AUTOGRAPHIC VISUALIZATIONS 101

Fig. 3. Autographic visualizations and their design operations (Table 2), top-left to bottom-right: (a) Cyanometer, a device for measuring the blueness
of the sky, framing and encoding [14]; (b) Mercury-in-glass thermometer, constraining and encoding; (c) filter for sampling airborne particulate
matter, aggregating; (d) southern blot for DNA electrophoresis, separating, registering; (e) EJ Marey’s smoke machine to visualize airflow, coupling;
(f) Campbell-Stokes sunshine recorder, registering, encoding; (g) Chladni figure revealing sound waves, coupling, registering; (h) a planning diagram
for neuro-surgery, annotating [81]; (i) pedocomparator for sampling and comparing soil samples, aggregating, encoding [37]; (j) reagent strips for
ozone detection, registering, encoding.

have a short symbolic distance, but a long causal distance because
of the complexity of the mediating apparatus. In place of these two
dimensions, the concept of qualitative displays elegantly presents a
one-dimensional measure of “directness,” describing the degree of
intervention by a designer [39]. This dimension spans five different
levels ranging from visual phenomena that are their own visualization
to highly artificial data physicalizations at both extremes of the scale.
The authors argue that visualization, so far, has been biased towards
quantitative information while neglecting qualitative aspects.

Indexical visualization and qualitative displays both are motivated
by a gap in existing frameworks: the neglect of the index compared to
icons and symbols in the former, the neglect of qualitative information
in the latter. Both emphasize the embeddedness of visualizations in
the physical world [78]. Neither, however, fully capture the nature of
analog visualizations of material information: Indexicality requires
adhering to a semiotic framework that insists on explicating visual
codes. The term qualitative display, on the other hand, seems overly
broad as a descriptor of material displays. The term autographic ad-
dresses the main difference to information design, InfoVis, and data
physicalization: the self-inscribing nature of material displays, in which
the designer creates the apparatus that lets traces emerge rather than
explicitly defining symbolic mappings. Areas of intersection exist: for
example, data visualization software that generates and displays its own
data from user interaction and therefore assumes autographic qualities,
or projects such as Dear Data, when the signature of the author is
considered as a trace [41].

Autographic visualization continues the explorations into self-
illustrating phenomena, first presented by Pat Hanrahan in his 2004
IEEE Vis capstone talk [24]. Referencing a concept from the his-
tory of scientific representation [59], Hanrahan focused on scientific
experiments rather than the broader cultural field of visual practices.
Autographic displays, however, are not limited to science but can be
found throughout history and culture. The term autographic not only
reflects the process of visualization and the role of the designer in this
process but is also historically accurate, since the term was widely used
during the late 19th and early 20th century to describe self-inscribing

mechanisms [64]. As reflected by a Google n-gram search, the terms
“autographic” and “self-registering” saw their peak in the early 20th cen-
tury, where they show up in many patent applications for mechanical
visualization devices and photographic techniques, before losing popu-
larity later in the 20th century.1

5 AUTOGRAPHIC DESIGN OPERATIONS
The production of interpretable traces is facilitated by cultural tech-
niques that involve various degrees of intervention. In the most simple
case, an environmental trace presents itself to a skilled observer. At
the other end of the spectrum, traces are the product of a complex
experimental apparatus involving many transformations. Along this
continuum, the engagement with traces can be articulated as a design
process that comprises a set of operations to turn a phenomenon into
encodable data. The designer has to decide which aspect of a phe-
nomenon can be used as an indicator and proceed to apply different
operations that make this indicator legible.

The visual vocabulary of information visualization is formalized
in schemata ranging from the foundational concept of visual vari-
ables to the grammar of graphics, organized by data structure and
user needs [6, 9, 63, 76, 77]. A taxonomic approach that categorizes
trace-phenomena into visual variables seems impractical and would
introduce another level of symbolic representation. Instead, our ap-
proach focuses on the design operations involved in autographic design
(Fig. 3). Table 2 provides an overview of these operations, grouped
by the kinds of transformations they achieve. Literature categorizing
traces exists in domain-specific areas, from the forensic analysis of
crash skid marks [66] to the identification of animal tracks [38]. But
to our knowledge, there are no overarching accounts that generalize
the visual operations of trace-making across disciplines. The following
taxonomy is an attempt to this effect.

The construction of the proposed autographic design space involved
multiple steps. The fundamental concepts were drawn from theoret-

1See …

1077-2626 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 1, JANUARY 2020 119

What is Interaction for Data Visualization?
Evanthia Dimara and Charles Perin∗

Abstract—Interaction is fundamental to data visualization, but what “interaction” means in the context of visualization is ambiguous
and confusing. We argue that this confusion is due to a lack of consensual definition. To tackle this problem, we start by synthesizing
an inclusive view of interaction in the visualization community – including insights from information visualization, visual analytics and
scientific visualization, as well as the input of both senior and junior visualization researchers. Once this view takes shape, we look at
how interaction is defined in the field of human-computer interaction (HCI). By extracting commonalities and differences between the
views of interaction in visualization and in HCI, we synthesize a definition of interaction for visualization. Our definition is meant to be
a thinking tool and inspire novel and bolder interaction design practices. We hope that by better understanding what interaction in
visualization is and what it can be, we will enrich the quality of interaction in visualization systems and empower those who use them.

Index Terms—interaction, visualization, data, definition, human-computer interaction.

1 INTRODUCTION
The notion of interaction has been a challenging concept to define in
the field of Human-Computer Interaction (HCI). Only recently, an HCI
review entitled “What is interaction?” [56] summarized concepts that
describe the causal relationships between the human and the computer.
While this HCI review outlines opportunities for enriching interactivity
with computer systems, it is unclear how these concepts relate to visu-
alization. Therefore, while visualization researchers and practitioners
may be aware of such interaction concepts, they do not necessarily see
how to apply them to their own data-oriented practices and needs.

Meanwhile in the past decade we have witnessed a growing call
for enriching interactivity in visualization systems. Forward-looking
research on interaction for visualization advocate for visualization
systems that give absolute freedom to end users to actively restruc-
ture [73, 100], sketch [75], author [69, 119] and personalize [57, 111]
visualizations; to construct visualizations from scratch [59], perform
data-aware annotations on them [54, 117], and unruly remove distract-
ing information [29]; to enrich visualizations with external knowl-
edge [118], control fluently [36] both data presentations [101] and data
pre-processing statistical functions [37]; to indicate uncertainty [82],
collaborate with peers [55, 80], and interact with visualizations using
natural means [66, 74] within physically situated settings [63, 73, 121].

We argue that a strong barrier to achieving this vision is not only that
of the technical challenges, but like in HCI, that of defining interaction
for visualization. In the visualization pipeline [16], interaction occurs at
all stages of the visualization process of turning raw data into views on
the data. While the visualization community has iteratively structured
and formalized the representation aspect of the pipeline, significantly
less attention has been paid to the interaction aspect [36, 72, 74, 99].
The nature and role of interaction has actually sparked discussions and
arguments since the visualization field was created. As of today there is
no consensus on what interaction is, and what its role for visualization
is – as interaction is an elusive and overloaded term [74, 92, 126].

To address this problem, we first capture the current view of interac-
tion from the visualization community based on the input of researchers.
Once this view takes shape, we revisit the view of interaction from the
HCI community [56] to understand how the two views differ and relate
to each other. Combining these two perspectives, we then propose a

Evanthia Dimara is with Sorbonne University. E-mail:
[email protected]
Charles Perin is with University of Victoria. E-mail: [email protected]

∗All authors contributed equally to this manuscript, with the exception of the
first author who did most of the work.

definition of interaction for visualization. This definition attempts to
broaden the scope of interaction in visualization and is inclusive as it
considers the perspectives of information visualization, visual analyt-
ics, and scientific visualization. We further extend this definition to
operationalize flexibility within visualization systems, based on where
interactions occur semantically in the visualization pipeline. We hope
that this definition will spur novel, bolder interaction design practices
in visualization, and the growth of more flexible visualization tools.

2 A REVIEW OF INTERACTION FOR VISUALIZATION

To capture the view of interaction in visualization, we started with a list
of papers based on our own expertise and expert input. Then we applied
a recursive process to expand our review while accounting for our own
bias. We describe our methodology rationale, method for collecting
papers, questionnaire we sent to experts, paper collection method, and
tagging method. Then we present the summary statistics of the review.

2.1 Methodology Rationale

The topic of interaction in visualization is broad; arguably most visual-
ization papers mention interaction at a point. Thus we discarded the
systematic review and instead opted for a critical review, that needs not
be exhaustive but requires a more detailed examination of the litera-
ture [46]. Because critical reviews seek to identify the most significant
items in the field [46], we set the three following requirements:

R1:: The view of interaction of the community cannot be captured only
by citation number, it needs to include expert input.

R2:: The snowballing approach alone (starting with a set of seed papers
and expanding using back- and forward-references) is not sufficient
because it is biased by the selection of seed papers.

R3:: The review must not include HCI papers unrelated to visualization.
We used two metrics to measure the impact and relevance of a paper

P based on publication year Py. IIImmmpppaaacccttt PPP = (Pc/10)/(current year −
Py + 1) measures the importance of the paper to the community based
on its number of citations Pc. RRReeellleeevvvaaannnccceeePPP = Pf /(current year −Py +
1)+(Pb + 1)/(Py −1980 + 1) measures the relevance to the topic of
interaction based on its number of forward references Pf and backward
references Pb (i.e., papers that cite, and that are cited by, P) that contain
both “interaction” and “visualization” in their title. 1980 is the year of
the oldest paper [17] we collected. These metrics identify both highly
relevant and impactful papers, while not relying solely on bibliometrics.

We set the inclusion criteria to ImpactP > 0.5, keeping papers with
roughly more than 10 citations a year, and to RelevanceP > 0.2, keeping
papers with roughly more than two forward or backward references
with the keywords “interaction” and “visualization” in their title every
10 year. These cutoffs (which are broad to prevent false negatives)
include important papers but exclude some clearly non-relevant ones.

Manuscript received 31 Mar. 2019; accepted 1 Aug. 2019.
Date of publication 16 Aug. 2019; date of current version 20 Oct. 2019.
For information on obtaining reprints of this article, please send e-mail to:
[email protected], and reference the Digital Object Identifier below.
Digital Object Identifier no. 10.1109/TVCG.2019.2934283

Authorized licensed use limited to: Grand Valley State Univ. Downloaded on January 27,2022 at 23:17:15 UTC from IEEE Xplore. Restrictions apply.

120 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 26, NO. 1, JANUARY 2020

2.2 Review Questionnaire
We sent visualization researchers an online form asking for: (1) defini-
tion papers, the papers that attempt to define interaction in visualization;
(2) relevant papers to the topic of interaction; (3) their years of visu-
alization experience; and (4) their interaction experience on a 7-point
scale describing how often their own papers focus on interaction. There
were also three optional fields: name, email, and comments.

2.3 Method for Collecting Papers
We created a list of seed papers with the following snowballing method:
Step 1: We started from our own list of 5 definition papers (R1).
Step 2: We sent the questionnaire to expert visualization researchers

who suggested both definition and linked papers (R1,R2).
Step 3: We included in the list of seed papers each paper P that:

was included in our initial list of 5 definition papers or was
suggested as a definition paper at least once; and
is a journal article or conference paper – to avoid non-peer
reviewed entries such as demos; and
has the term “visualization” in the title or abstract (R3); and
has RelevanceP > 0.2 and ImpactP > 0.5.

In addition, considering that expert input can provide insights not
captured by our computational method (R1), we included in the
list of seed papers those that did not fulfill these inclusion criteria
but that were suggested as definition paper three times or more.

Step 4: We repeated Step 2, sending the questionnaire to all (non-
previously contacted) authors of the added seed papers. We
then repeated Step 3 with the new suggestions. We stopped the
recursion when no new seed paper or author was found.

Once the list of seed papers finalized, we derived a set of linked papers.
A paper P is linked if it is not seed and one of the following is true:

P is suggested by two or more experts as definition or relevant;
P cites or is cited by a seed paper and its title contains both
“visualization” and “interaction”.

2.4 Method for Tagging Papers with Interaction Themes
We (the two authors of this paper) developed tags (short labels) to
characterize the seed and linked papers using an iterative deductive
coding method [88]. Specifically, we identified emerging themes from
reading seed papers with the lens of capturing: i) the current view of
interaction in visualization; ii) problems raised by the community; and
iii) existing archetypal descriptions and categorizations of interactions.
We then refined these tags incrementally until we obtained a high
inter-coder agreement. We then coded all seed and linked papers. We
measured inter-coder agreement using Cohen’s Kappa [24] (K), that
accounts for the possibility that agreement occurs by chance. There is
no strict rule regarding how to interpret values of K, but inter-reliability
is considered satisfactory for K ≥ 0.7 and excellent for K > 0.75 [41].

For the first coding iteration, we used plain tag names. We indepen-
dently tagged 6 seed papers using the tag names described below with
binary values (a paper either has the tag or not). We obtained K = 0.57.
We discussed similarities and differences in our tagging, then wrote the
detailed definitions for each tag presented below. We switched to three
values: 0 (not applicable tag), 1 (partial fit of the tag), and 2 (good fit
of the tag). For the second iteration, we independently coded again
the 6 first seed papers (K = 0.83), as well as 6 additional seed papers
(K = 0.78), showing excellent agreement (K > 0.75). For the third
iteration, we independently coded 12 new seed papers. We obtained
K = 0.87, which allowed us to have a single coder per remaining paper.
Tag — Definition of Interaction: attempt to define, explain or de-
scribe interaction. A rating of 1 marks implicit attempts, including
definitions based on specific properties or attributes of interaction. A
rating of 2 marks direct definitions (formal or informal, e.g., “interac-
tion is …”).

Tag — Critical on Interaction: complaints and frustrations (using a
negative tone) about interaction, to reveal areas of improvement and
challenges. A rating of 1 marks brief complaints. A rating of 2 marks
extensive discussions of complaints.

Tag — Benefits of Interaction: positive aspects of interaction (actual
or expected). A rating of 1 marks brief (or moderately phrased) dis-
cussions of benefits. A rating of 2 marks more elaborated (or strongly
phrased) discussions of benefits.

Tag — Interaction Concepts: archetypal description of interaction.
Given to papers that contribute one of: “concept”, “model”, “frame-
work”, “design space”, “paradigm”. A rating of 1 marks concepts with
small or unclear relation to interaction. A rating of 2 marks concepts
with interaction as a key element (e.g., an interaction model).

Tag — Interactive Pipeline: discussion of interaction in relation to
the visualization pipeline, a fundamental concept that transcends sub-
domains of visualization and explicitly includes interaction. It is given
to papers that use the word “pipeline” or cite one of [16, 20, 21, 48]. A
rating of 1 indicates that the existence of interaction is unclear. A rating
of 2 indicates that the interaction component is clearly discussed.

Tag — List: contribution of a list (e.g., of interaction techniques, tasks,
or intents). We call formal a list that is described with one of the follow-
ing words: “taxonomy”, “classification”, “typology”, “categorization”.
A rating of 1 marks an informal review (it does not contain any of the
formal keywords). A rating of 2 marks a formal review.

2.5 Summary Statistics of the Review1

RECRUITING: We emailed 77 visualization experts over a two-month
period. 64/77 were successfully delivered. 34/77 were in our initial
list of experts and the remaining 43/77 were derived through our re-
cursive algorithm. While we started with a bias toward information
visualization (e.g., [36, 72, 79, 106]), our algorithm expanded the scope
naturally to visual analytics (e.g., [37, 45, 50, 92]), scientific visualiza-
tion (e.g., [1, 66, 67, 77]) and digital cartography (e.g., [98, 99]).

PARTICIPANTS: 22 visualization researchers responded to the survey.
They had 5–32 (M : 13.6, SD : 6.2) years of visualization experience.
12/22 were authors of seed papers, and 14/22 were authors of linked
papers; 2/22 completed the survey anonymously. Participants rated
their interaction expertise with a mean of 5.9/7 (SD : 0.8).
SEED PAPERS: Fig. 1 presents all 59 tagged papers. We started with
5 definition paper based on our expertise. Participants suggested 114
(83 unique) definition papers (6 did not suggest any definition paper).
Applying our recursive algorithm resulted in a set of 23 seed papers.
LINKED PAPERS: Participants suggested 61 (51 unique) relevant papers.
Forward and backward searches gave 105 papers citing, and 104 papers
being cited by a seed paper. Our linked paper collection algorithm
gave 47 linked papers. We excluded 7 theses, 3 unpublished and 1
non-English reports. This resulted in 36 linked papers.
TAGS: The derived tags were: definition: 32; benefit: 39; critical: 36;
concept: 20; pipeline: 12; and list: 35.

3 THE VISUALIZATION VIEW OF INTERACTION
In this section, we synthesize the current view of interaction in visu-
alization based only on the points of view of the 59 seed and linked
papers we reviewed (see Fig. 1). We elaborate on our own critique of
this literature in Sect. 4 and Sect. 5. While we reviewed all 59 papers,
due to space limitations, we only cite papers which have been cited
more than twice – and provide all papers in supplementary material.

3.1 Definitions of Interaction for Visualization
Interaction has been identified as an overloaded [92] and elusive [36,74]
term, and it is challenging to find a solid definition of interaction
[36, 74, 126]. Therefore, we consider the 32 papers with a definition
tag – whether they attempt to define interaction implicitly or explicitly.

3.1.1 What Interaction for Visualization Must Involve
The definition papers reveal mandatory components of interaction,
namely: external and internal entities, external and internal actions.

EXTERNAL ENTITIES: The two most cited mandatory entities are
the user and the data. The user (or analyst [8, 37, 112]) is a human

1 Material: https://osf.io/ej7xg/?view only=51485163dfc94d0c8499af17cb2038b2

Authorized licensed use limited to: Grand Valley State Univ. Downloaded on January 27,2022 at 23:17:15 UTC from IEEE Xplore. Restrictions apply.

DIMARA AND PERIN: WHAT IS INTERACTION FOR DATA VISUALIZATION? 121

THE INFORMATION VISUALIZER, AN INFORMATION WORKSPACE
THE EYES HAVE IT: A TASK BY DATA TYPE TAXONOMY FOR INFORMATION VISUALIZATIONS.
ON THE SEMANTICS OF INTERACTIVE VISUALIZATIONS.
AN OPERATOR INTERACTION FRAMEWORK FOR VISUALIZATION SYSTEMS
A TAXONOMY OF VISUALIZATION TECHNIQUES USING THE DATA STATE REFERENCE MODEL
INFORMATION VISUALIZATION AND VISUAL DATA MINING
BEYOND MOUSE AND KEYBOARD: EXPANDING DESIGN CONSIDERATIONS FOR INFORMATION VISUALIZATION INTERACTIONS
INTERACTION SPACES IN DATA AND INFORMATION VISUALIZATION
AN INTERACTION VIEW ON INFORMATION VISUALIZATION
LOW-LEVEL COMPONENTS OF ANALYTIC ACTIVITY IN INFORMATION VISUALIZATION
TOWARD A DEEPER UNDERSTANDING OF THE ROLE OF INTERACTION IN INFORMATION VISUALIZATION
A FRAMEWORK OF INTERACTION COSTS IN INFORMATION VISUALIZATION
SPATIAL REASONING WITH EXTERNAL VISUALIZATIONS: WHAT MATTERS IS WHAT YOU SEE, NOT WHETHER YOU INTERACT
THE SCIENCE OF INTERACTION
CHARACTERIZING USERS’ VISUAL ANALYTIC ACTIVITY FOR INSIGHT PROVENANCE
MENTAL MODELS, VISUAL REASONING AND INTERACTION IN INFORMATION VISUALIZATION: A TOP-DOWN PERSPECTIVE
FLUID INTERACTION FOR INFORMATION VISUALIZATION
SEMANTIC INTERACTION FOR VISUAL TEXT ANALYTICS
INTERACTIVE DYNAMICS FOR VISUAL ANALYSIS : A TAXONOMY OF TOOLS THAT SUPPORT THE FLUENT AND FLEXIBLE USE OF VISUALIZ
AN EMPIRICALLY-DERIVED TAXONOMY OF INTERACTION PRIMITIVES FOR INTERACTIVE CARTOGRAPHY AND GEOVISUALIZATION
A MULTI-LEVEL TYPOLOGY OF ABSTRACT VISUALIZATION TASKS
INTERACTION DESIGN FOR COMPLEX COGNITIVE ACTIVITIES WITH VISUAL REPRESENTATIONS: A PATTERN-BASED APPROACH
A DESIGN SPACE OF VISUALIZATION TASKS
BRUSHING SCATTERPLOTS
STARTING SIMPLE: ADDING VALUE TO STATIC VISUALISATION THROUGH SIMPLE INTERACTION
READINGS IN INFORMATION VISUALIZATION : USING VISION TO THINK
ILLUMINATING THE PATH: THE RESEARCH AND DEVELOPMENT AGENDA FOR VISUAL ANALYTICS
A TAXONOMY OF TEMPORAL DATA VISUALIZATION TECHNIQUES ???? (NOT SURE HOW THIS PAPER APPEARED)
A PROPOSAL FROM THE POINT OF VIEW OF INFORMATION VISUALIZATION AND HUMAN COMPUTER INTERACTION FOR THE VISUALIZATI
INFORMATION VISUALIZATION : DESIGN FOR INTERACTION
VISUAL PERCEPTION AND MIXED-INITIATIVE INTERACTION FOR ASSISTED VISUALIZATION DESIGN
STACK ZOOMING FOR MULTI-FOCUS INTERACTION IN TIME-SERIES DATA VISUALIZATION
ADVANCED INTERACTION FOR INFORMATION VISUALIZATION.
INTEGRATING VISUALIZATION AND INTERACTION RESEARCH TO IMPROVE SCIENTIFIC WORKFLOWS
AN EXPLORATORY STUDY OF INTERACTIVITY IN VISUALIZATION TOOLS:’FLOW’OF INTERACTION
FI3D: DIRECT-TOUCH INTERACTION FOR THE EXPLORATION OF 3D SCIENTIFIC VISUALIZATION SPACES
EXPLORING INFORMATION VISUALIZATION: DESCRIBING DIFFERENT INTERACTION PATTERNS
ANALYSTS AREN’T MACHINES: INFERRING FRUSTRATION THROUGH VISUALIZATION INTERACTION
CARTOGRAPHIC INTERACTION PRIMITIVES: FRAMEWORK AND SYNTHESIS
AN INTERACTION MODEL FOR VISUALIZATIONS BEYOND THE DESKTOP
REIMAGINING THE SCIENTIFIC VISUALIZATION INTERACTION PARADIGM
MULTILEVEL INTERACTION MODEL FOR HIERARCHICAL TASKS IN INFORMATION VISUALIZATION
INTERACTION IN THE VISUALIZATION OF MULTIVARIATE NETWORKS
DECLARATIVE INTERACTION DESIGN FOR DATA VISUALIZATION
MOVEXP: A VERSATILE VISUALIZATION TOOL FOR HUMAN-COMPUTER INTERACTION STUDIES WITH 3D PERFORMANCE AND BIOMECHAN
AN INTERACTION FRAMEWORK FOR LEVEL-OF-ABSTRACTION VISUALIZATION OF 3D GEOVIRTUAL ENVIRONMENTS
ADAPTIVE VISUALIZATION INTERFACE THAT MANAGES USER’S COGNITIVE LOAD BASED ON INTERACTION CHARACTERISTICS
INTERACTION FOR VISUALIZATION SYNTHESIS LECTURES ON VISUALIZATION
TOWARDS THE UNDERSTANDING OF INTERACTION IN INFORMATION VISUALIZATION
QUERY2QUESTION: TRANSLATING VISUALIZATION INTERACTION INTO NATURAL LANGUAGE
NATURAL INTERACTION WITH VISUALIZATION SYSTEMS
A VISUALIZATION-ANALYTICS-INTERACTION WORKFLOW FRAMEWORK FOR EXPLORATORY AND EXPLANATORY SEARCH ON GEO-LOCATE
EVALUATION OF TWO INTERACTION TECHNIQUES FOR VISUALIZATION OF DYNAMIC GRAPHS
A CASE STUDY USING VISUALIZATION INTERACTION LOGS AND INSIGHT METRICS TO UNDERSTAND HOW ANALYSTS ARRIVE AT INSIGHT
FLEXIBLE ORGANIZATION, EXPLORATION, AND ANALYSIS OF VISUALIZATION APPLICATION INTERACTION EVENTS USING VISUAL ANALY
SPATIALVIS: VISUALIZATION OF SPATIAL GESTURE INTERACTION LOGS
IVORPHEUS 2.0-A PROPOSAL FOR INTERACTION BY VOICE COMMAND-CONTROL IN THREE DIMENSIONAL ENVIRONMENTS OF INFORMAT
VISUALIZATION BY DEMONSTRATION: AN INTERACTION PARADIGM FOR VISUAL DATA EXPLORATION
VISUALIZATION AND INTERACTION WITH MULTIPLE DEVICES. A CASE STUDY ON REACHABILITY OF REMOTE AREAS FOR EMERGENCY MA

AUTHORSTITLEYEAR
A
I

D

S
Z

M
E

H

I
N

O
I
A
S

O

E
Q
V

F
S
N
H
T

F
H
N
E
N
K
T

O
N
R

R
T

A
A

A
F
C

R
S

LOW HIGH

0 1 2

S
E

E
D

P
A

P
E

R
S

L
IN

K
E

D
P

A
P

E
R

S

TAGS

METRICS
TA

GS

M
ET

R
IC

S
CONCEPT

PIPELINE

RELEVANCE P

FORWARD PEXPERT INPUT

PC (citations)

IMPACT P

BACKWARD P
0

12

CRITICAL
BENEFITS

DEFINITION

1991
1996
1996
1998
2000
2002
2003
2004
2004
2005
2007
2008
2008
2009
2009
2010
2011
2012
2012
2013
2013
2013
2013
1987
1998
1999
2005
2005
2005
2007
2008
2010
2010
2010
2010
2010
2010
2011
2012
2013
2013
2013
2014
2014
2014
2014
2014
2015
2015
2015
2015
2015
2016
2016
2016
2016
2016
2017
2017

CARD ET AL.
SHNEIDERMAN

CHUAH AT AL.
CHI & RIEDL

CHI
KEIM

LEE ET AL.
WARD & YANG

KOSARA ET AL.
AMAR ET AL.

YI ET AL.
LAM

KEEHNER ET AL.
PIKE ET AL.

GOTZ & ZHOU
LIU & STASKO

ELMQVIST ET AL.
ENDERT ET AL.

HEER & SHNEIDERMAN
ROTH

BREHMER & MUNZNER
SEDIG & PARSONS

SCHULZ ET AL.
BECKER & CLEVELAND

DIX AND ELLIS
CARD ET AL.

COOK & THOMAS
DAASSI ET AL.

LARREA ET AL.
SPENCE

HEALEY ET AL.
JAVED & ELMQVIST

FEKETE
KEEFE

LIANG ET AL.
YU ET AL.

POHL ET AL.
HARRISON ET AL.

ROTH
JANSEN & DRAGICEVIC

KEEFE & ISENBERG
REN ET AL.

WYBROW ET AL.
SATYANARAYAN ET AL.

PALMAS ET AL.
SEMMO & DÖLLNER

YELIZAROV & GAMAY
TOMINSKI

FIGUEIRAS
NAFARI & WEAVER

AMANT
SJÖBERGH ET AL.

FEDERICO & MIKSCH
GUO ET AL.
HAN ET AL.

PALUKA & COLLINS
FURTADO ET AL.

SAKET ET AL.
TOST & HEIDMANN

LIST

Fig. 1. The 23 seed and 36 linked papers in our review, ordered by year.
It shows expert input, the algorithm metrics, and our tagging scores.

being who can be characterized by skills [36], abilities, expertise and
motivation [99], and who initiates the interaction [104]. Some papers
differentiate between end-users and designers [22]. The data is an
intangible information source that is the user’s main object of interest.

The mean with which the user interacts with the data is a mediating
[98, 99] entity that we call the visualization system. The visualization
system (either as a whole or some of its components) is referred to as
technology [74, 99, 104], computational tools [92] and computer [8]
with hardware and software [8], when emphasizing its technological
properties; representation [76, 104, 126], visualization [16, 19, 22, 45, 67,
68,72,79,99,120], map [98], graphical marks [2,22], display [8,74,106],
visual metaphor [37] and non-static image [70], when emphasizing its
visual properties; and interface [19, 22, 36, 76, 92, 98, 99, 104], system
[19, 36, 45, 62, 74, 123, 126], analysis tool [92], workspace [19] and
interaction space [104], when referring to a more abstract mediating
entity. Some papers add that the mediating entity should provide a set
of controls [92] for the user to access the data.

Most papers identify time as a fundamental entity of interaction [16,
62, 76, 126]. Interaction, unlike representation [126], is seen as a real-
time [8, 98] action with a start and an end [74] that should result in an
immediate response from the visualization system [16, 19, 36, 106, 123].

INTERNAL ENTITIES: Interaction is characterized as a goal-oriented
activity [53, 62, 92] involving a data-oriented intent. Intent [36, 45,
74, 76, 79, 84, 92] is also phrased as goal [45, 62, 76, 92, 97–99, 123],
task [45, 67, 106] or problem [53, 92]. Intent can describe a high-level
data exploration [19, 45, 62, 68, 84, 92, 126], the generation of insights
[45, 62, 92, 98, 99, 126], and the need to acquire multiple perspectives
on the data [62, 92, 104, 126]. Intent can go beyond exploration, such as
the intent to collect and correct data [62], or social intents to coordinate
in collaborative setups and to present data to an audience [62]. Intent is
both identified at a low, operational level, e.g., to alter the representation
[19, 126], and at a higher level, e.g., for information foraging [79],
sensemaking [99, 104], and knowledge creation [92].

EXTERNAL ACTIONS: Interaction is described as a dialogue between
the user and the visualization system [36, 76, 92, 98, 104, 123, 126] made
of action-reaction pairs [76, 104]. The user performs an action (or
input [22, 102]) on the visualization system [45, 67, 72, 76, 79, 104]. The
visualization system returns a reaction [74, 76, 104] (or response [19, 67,
74,76,98,106,120,126], change [8,16,22,98,120,126], output [22,102])
that is perceived by the user [72, 104, 120]. This reaction has been
called permutation of graphical marks [2], change of transformation
parameters [123], and alteration of the pipeline [62].
INTERNAL ACTIONS: Along with its physical acts, interaction with a
visualization system involves a cognitive act of the user [37, 92, 104] or,
similarly, a reasoning/analytic process [67, 76, 79, 97] on the data.

3.1.2 What Interaction for Visualization Can Involve
Interaction for visualization can involve additional external and internal
entities, and additional external and internal actions.

EXTERNAL ENTITIES: Interaction can involve external physical objects
[62, 74, 106] such as mouse [79], pen [74] keyboard [79] and physical
constraints [99]. It can also involve a variety of modalities such as
body movements [19, 74], speech [74], head [79] and eye movement
[79]. Many modalities (e.g., gaze, head) tend to be overlooked in
visualization. Restricting modalities leads to a “disjointed picture of
human performance” [79] and lost opportunities to capture user intent
[79]. Interaction also involves the environment under which interaction
occurs, for example whether it is a casual or working environment [99],
and whether there are multiple users involved [62, 74, 92, 99].

INTERNAL ENTITIES: Users’ prior knowledge [92], internal mental
representation [104], skills [36] and abilities [99] can be involved when
interacting with a visualization system. While interaction can start
with a concrete user query [19, 45, 84, 106], it can also lead to the
internalization of new goals [104]. Moreover, interactions might occur
with an absence of intent, such as with proxemics interactions [74].

EXTERNAL ACTIONS: Interaction can involve many user actions. Such
action can change the data [16, 22, 45, 76, 84, 92, 102, 104], e.g., with
filtering and aggregation [102]. It can change the representation and
presentation of information [8, 16, 19, 37, 45, 68, 79, 84, 104], e.g., with
sorting [45] or when switching from a map to a timeline [45]. It
can create metadata, for example by temporarily marking data to track,
annotate or bookmark [37,45,84]. It can create new data representations
[74, 79] and new data, to express and manipulate new knowledge [37,
45, 104], e.g., with note taking and when manipulating a knowledge
management component [45]. A user can perform an action that does
not occur on the data or their representation, e.g. adjusting a movable
baseline to compare the heights of a histogram [126], performing a
metaction on their own action history, (undo/redo) [45], and steering a
statistical data model [37]. Last, along with the dialogue between the
user and the visualization system, there is also the dialogue of users
with the external environment [92] and with their peers [62, 74, 92, 99].

INTERNAL ACTIONS: While interaction with a data source is enabled
within the context of a tangible visualization system, much of it can
occur internally in user’s mind [74, 92], for example information pro-
cessing [104], memory encoding [104], and simulative reasoning [79].
On a deeper level, the interaction takes the form of a dialogue between
the “internal representations and processes of the user and the external
representations and processes” of the visualization system [104].

3.1.3 Properties of Interaction
Interaction for visualization is characterized as a goal-oriented [62,
76, 92, 97–99, 123] activity that contains semantics [22, 45], that is
sequential [22, 45, 98, 104], incremental [16, 36, 37, 106], and iterative
[19, 98, 104], and preserves the following properties:

GRANULARITY: Interaction is characterized at multiple levels of gran-
ularity [2, 45, 62, 76, 102, 104]. Such levels include micro-level (e.g.,
mouse clicks), macro-level (e.g. hypothesis generation) and levels in
between (e.g., filter, sort). To distinguish these …