The cognitive
side of Visual Languages
by Ketil Hunn
Course: INFSCI
2650 Visual Languages
Course instructor: Dr.
Robert Korfhage
Since the introduction of linear and symbolic
computer languages over 40 years ago people have striven
for a more visual way of programming computers. This
paper will focus on the cognitive side of visual
languages and how we can use this knowledge to better
create visual interfaces and visual languages. The paper
will in particular deal with three papers submitted at
the Visual Languages Conference in 1986 by Jacob, Tauber
and Chu, and an article by Montalvo 1990.
1. Introduction
Although the symbolic programming language was
introduced over 40 years ago it is still the most widely used
tool to produce computer programs, it was in fact the only way to
program up until the 1980’s. But still we tend to use an
iconic representation when we explain an algorithm or a piece of
code to other people (Jacob 1986). One might think that with the
powerful computers we have today we would use a more graphical
way of producing programs, but in fact the way of programming has
not changed much since the introduction of programming languages.
Even if the concepts behind visual languages were introduced in
the 1980’s, visual languages are still far from complete and
they are not yet a competitor to symbolic programming languages.
This paper will briefly focus on the difficulties we face when
implementing visual languages and in particular deal with the
cognitive side of interface design.
2. What is a visual
language
A visual language might best be described as
language that deals mostly with graphical symbols instead of
symbolic text. In contrast to the symbolic way of programming,
where the symbols are interpreted linearly, visual languages will
allow the user to place objects in a n-dimensional space. The
type of graphical objects and how they are combined and placed in
this space will make a difference on the program design. Being
able to design and create computer programs merely by using
graphical objects will make it easier for novices to create
programs without having to learn the somewhat abstract and
unnatural way of symbolic languages.
The figure to the right gives an example of a simple program
that adds 1 to a variable every time the user press the button.
As illustrated, visual languages do not consist entirely of
graphical symbols, because labels can be used to clarify the
meaning of the graphical symbols. And that, as we will see later,
is the very heart of the problems with visual languages: the
ambiguity of the icons used in visual languages.
Using a visual language the user will be able to describe a
program without knowing how the program will be implemented, in
other words the functions will be separated from the
implementation (Jacob 1986). And by allowing the user to draw his
or her mental model of the program instead of thinking about the
implementation would certainly take less effort to produce a
program than with the conventional way of programming (Jacob
1986).
3. The graphical objects
of a visual interface
A visual language consists of a set of basic graphical objects
and a set of operations or methods that you that you can perform
on them. The graphical objects should be designed in such a way
that it is apparent how they can be combined. In some systems
this is solved by attaching hooks to the graphical objects so
that only objects that it makes sense to combine can be combined,
just like a jigsaw puzzle. Others visual languages strive for a
good design of the graphical objects so that it will be apparent
to the users which objects can be combined. Yet another method is
to implement a smarter system that will try to figure out what
the user wants to do.
Chu (1986) stress that the design of the graphical symbols
should be kept as simple as possible instead of having a high
information density. Having a set of simple base icons will allow
the user to combine the simple ones into more complex objects,
just like a programmer can build complex functions out of simple
building blocks in a symbolic language. Chu also stress the
importance of images having universal comprehensibility so they
can be recognized by many. Montalvo (1986) also points out the
importance of the system giving meaningful feedback and that this
will stimulate the human interaction with the system Only when
these conditions are met can a visual language be useful.
Chu also mentions the need for a hierarchic structure and
concepts. A hierarchical structure will allow similar objects to
be grouped together and a concept will describe the attributes
and relationships within the group. According to Ausubel’s
theory (as referred to in Chu, 1986) there are only two basic
types of learning, namely learning by rote, where new information
is added to your existing knowledge without establishing any new
relationships with the existing knowledge, and meaningful
learning, where the new information is added to the existing
knowledge and relationships between the new and existing
information are created.
This is very important if the visual language contains a lot
of objects, because then the user will be more likely to forget
the meanings of some symbols. With a hierarchical structure the
user can more easily retrieve the meaning of a symbol based on
the attributes common to other symbols in the same hierarchical
group. If the symbols are well designed and arranged in a
well-form hierarchy the user will be more likely to remember the
individual symbols, even if the number of symbols are great. Chu
(1986) uses the Chinese language as an example. Even though there
is said to be several thousand different characters in use today,
they impose little or no problem because the characters are
organized hierarchically and the users use concepts and context
to more rapidly find the character or meaning they want. Chi also
points out that knowledge tend to be organized hierarchically and
that the learning process is more successful when more inclusive
concepts are introduced first. This is an important idea to keep
in mind when designing a visual language.
Jacob (1986) and Chi (1986) divide the objects of a visual
language into two categories, the first being a static graphical
object, such as screen layout or menus. These objects are easy to
incorporate into a visual language, because they represent
something concrete in the language, such as a procedure or a
sequence of functions. These graphical objects can be moved and
otherwise manipulated by the user to edit the program. The second
category represents more of a problem for a visual language: How
to represent the more abstract elements of a programming
language, such as time sequence and conditional statements. These
elements do not have obvious graphical symbols and they are not
easy to represent on screen as something that can be manipulated
directly.
So how does one represent these elements with graphical icons
which are unambiguous to the users? To be able to choose a good
graphical representation of the abstract elements of a computer
programming language one need to understand how humans perceive
graphical information. It is also important to introduce the
graphical objects in a meaningful context and environment.
4. Human perception of
graphics
Preattentive processing of visual information is performed
automatically on the entire visual field detecting basic features
of objects in the display. Such basic features include colors,
closure, line ends, contrast, tilt, curvature and size (Treisman,
1986). These simple features are extracted from the visual
display in the preattentive system and later joined in the
focused attention system into coherent objects (Treisman 1985).
Preattentive processing is done quickly, effortlessly and in
parallel without any attention being focused on the display.
Treisman (1986) also found support for the theory that some
properties are coded as values of deviation from a null or a
reference value. Thus, the more features an object has, the more
likely it will pop out.
Jacob (1986) also demonstrated this with an experiment in
which the subjects where given a set of 50 points in a
nine-dimensional space which were to be organized into five
groups. The subjects were given three different representations
of the data, one where the data was represented by a matrix of
nine digits, one where the data was represented by polygons and
finally a representation where the data was represented by
Chernoff faces. The subjects performed significantly better with
the Chernoff representation than with the other two
representations. In addition to the Chernoff representation
having more features than the other two representations, Jacob
(1986) also talks about the coherent gestalt effect where the
recognizable features of a face encourages people to combine the
data into a single memorable expression.
Still, with these results in mind, it is not easy to design
graphical representations that will work for all users. In fact,
Montalvo (1986) states that graphic interfaces cannot possibly
work for everybody at all times. The icons represent something
and what the designer had in mind may not be how the user
interprets it. Icons are ambiguous by nature and the user has to
know the context and environment in which the icon appear
(Montalvo, 1986).
5. Designing a visual
interface
When designing a visual interface the designer should first of
all consider the needs for the system. How will people perceive
and think about the system (Tauber 1986)? What will it be used
for? Although the Chu article stress the importance of well
designed graphical objects it also highlights the importance of
syntactic, semantic and pragmatic rules for the objects. This
view is also supported by Jacob (1986). Both divide the design of
the interface into three levels to reduce the complexity of the
designing process. The semantic level describes the meaning of
the program which acts as an abstraction from the functions of
the system. The syntactic level describes how the objects can be
combined together to form valid meanings and finally the lexical
level describes how the input and output is represented to the
user.
Tauber (1986) goes into even more detail when describing how a
visual interface may be implemented. Not only does he describe
the objects, but also the methods and even the functions
associated with the objects. His description of the user’s
virtual machine (UVM) and its objects sound surprisingly similar
to an object oriented programming language (OOP) even if it is
commonly accepted today that an OOP is not a visual language. It
contains many of the same elements such as hierarchy and
structure, but it lacks the visualization of objects. Programming
environment such as the Visual C++ compiler is not considered to
be a visual language, but rather a front end code generator.
6. The future of Visual
Languages
After viewing the different aspects and obstacles of implementing
a visual language, does it seem to have a future at all? It is
hard to say definitely yes or no to this question. All four
articles stress that a visual language is hard to implement and
that we still have a long way to go before we will see a visual
language that can be really useful. It seems rather impossible to
create a set of graphical icons that will be obvious to all users
at all time. Graphical objects can be interpreted in an unlimited
number of ways and that can in fact make visual languages appear
even more difficult to the programmer than a linear, symbolic
language. For example, how could recursive calls of an object be
visualized? Since the user should be able to combine objects,
these objects should also be able to call itself recursively.
Simply having an arc to itself will not be sufficient, because
the user will not know if the object is going deeper into the
recursion or if it is going the other way. Some implementations
of a visual language allow the user to double click on an object
to go deeper into the recursion, but how do you visualize that
the recursion is done and the control is going back out of the
recursion?
To overcome the ambiguity of graphical symbols the visual
language needs some kind of artificial intelligence and I am
surprised that none of the four articles even mention this. With
artificial intelligence the visual editor could make an educated
guess of what the user really want and act accordingly. A visual
language should also be flexible enough to handle ambiguous
actions from the user. For example, is dropping a spreadsheet
object on a calculator the same as dropping a calculator object
on a spreadsheet object?
Visual languages certainly seem like an useful tool to
visualize programs, especially showing the flow of a program. It
may perhaps be used as an introduction to symbolic languages and
especially demonstrate the importance of having a mental model of
a program before you start implementing it.
Instead of thinking of visual languages as a future
replacement of general purpose programming languages, we may have
to let it evolve and see if we can find useful niches for it. A
visual language is totally different from any linear, symbolic
language and perhaps will visual languages in the future allow us
to do things that were not possible with today's languages or
perhaps do it in a way that will be more obvious and intuitive
for non-programmers.
References
Chu, Kai. (1986). The cognitive aspects of Chinese character
processing. Visual Languages/1, 349-392.
Jacob, R.J.K. (1986). A visual programming environment for
designing user interfaces. Visual Languages/1, 87-107.
Montalvo, F.S. (1990). Diagram understanding: The symbolic
descriptions behind the scenes. VLA, 5-27.
Tauber, M.J. (1986), Top-down design of human-computer
interfaces. Visual Languages/1, 393-429.
Treisman, A (1985). Preattentive processing in vision.
Graphics- and image-processing, 31(2), 156-177.
Treisman, A. (1986). Features and objects in visual
processing. Scientific American, 255(5), 114B-125.