The design of visual languages

by Ketil Hunn

Course: INFSCI 2650 Visual Languages
Course instructor: Dr. Robert Korfhage

 

---

In the last two decades we have seen an increased demand for easy to use and intuitive programming environments. A great number of systems have been proposed, but very few have gained popularity. This paper will discuss some of the systems that have been introduced and why they have failed. The paper will then highlight important aspects of the design phase of visual languages and see how we can use this knowledge to design better visual languages.

1 Introduction
Since the dawn of computer science humans have striven for more intuitive and graphical programming environments than the conventional, linear, text-based systems, but it wasn’t until recently such systems could be realized, mainly because of the rapid increase in availability of sophisticated graphical microcomputers.

A visual language has several advantages over a text-based programming language. By presenting elements of the programming language graphically on screen such systems have the potential to be more intuitive and easier to understand than the text-based systems and by allowing the programmer to draw his or her mental model of a program directly on the computer screen we rapidly decreases development time as well as the time needed for novices to learn a programming language. A visual language will also be of great use for persons with reading-disabilities or persons that have problems with English in general since most text-based programming languages that exist today tend to be based on the English language.

But still with the graphical powers of computers today we tend to use the text-based languages. For, as we will see, creating good visual representations for objects and procedures turned out to be more difficult than we had thought. Visual programming environments are complicated systems which stress the importance of an effective user interface (McIntyre, Glinert, 1992) and their implementation has many of the software engineering problems well known in large systems. For a visual language to be truly effective the system has to be properly structured and designed. This paper will highlight the important aspects of the design of iconic visual languages based on papers submitted at Visual Language Conference in the period 1986 to 1992.

2 The knowledge behind visual languages
Tanimoto and Glinert (1986) mention three major categories that need to be addressed before an effective iconic language can be implemented. The designers need to model how people look upon computer programming and how they interact with computers. When fully understanding these concepts the designers will more easily be able to create an iconic language that will be easy to learn. The designers must also address the implementation problems that the graphical display itself will place upon the language, such as resolution, efficiency and responsiveness which are all important elements of the visual interface. The categories apply of course to all software development, but they are particularly important in developing iconic languages because the pictorial elements of an iconic language are by nature ambiguous.

In order for a visual language to be immediately understood by the user, its pictorial elements need to be based upon a model that is already well understood to avoid ambiguities, inconsistencies and overlapping of semantics (Cinque et all, 1990). By formalizing the design methodology we take an important step towards understanding iconic languages and communications (Cinque et all, 1990, Chang et all, 1992).

In the next sections some of important aspects of visual languages will be described.

3 Conceptual ladders
The Tanimoto and Glinert (1986) paper points out that one of the biggest problem with computer programs is that they often seems too abstract for novices and at the same time too limited for the experts. In order for a visual language to be usable for both novices and advanced users the language needs to grow with the novice until the novice becomes an expert. Tanimoto and Glinert (1986) suggest the idea of conceptual ladders as an important element of a visual language. At each level of progression, the visual system should provide the possibility of interaction between the user and the particular part of the system that is appropriate to level of the user. This will allow the novice to create programs even at the beginner’s level without having to know all of the elements of the language. The same idea can be found in textual languages, the user doesn’t necessarily have to know about functions and local variables to create a program that can be run, but it would be of great help if one wants to create an effective program.

4 Graphical representation of a program
An iconic language needs some way to intuitively display the structure of the program. This has the advantage over the textual based programming languages that the user can more easily and quickly extract information about the program’s structure and purpose (Tanimoto and Glinert, 1986).

The most commonly used representation today is flow of control diagrams, which shows the currently active state of the program (Tanimoto and Glinert, 1986). With this method the user can easily trace the execution of the program, and this has proven to be very valuable for novices helping them get a better understanding of the program. Illustrating the flow of a program in this way seems natural to humans and designers have been using static flowcharts for years. Using this method in visual languages we now have the possibility to put life into these flowcharts and make them dynamic. Research done in the field indicates that novices learn computer programming much more easily when a graphical display of the computations are shown on screen during runtime (Tanimoto and Glinert, 1986).

The problem with the flowchart approach however, is that the structure of the program needs to be mapped onto a 2 dimensional screen which is not always easy, especially when it comes to illustrating recursive functions. The user can easily lose track of scopes during recursive calls and it is the task of the designers to visualize this in an intuitive way.

If the program makes use of the flow of control model, the environment needs to zoom in on the active state and there is a trade off here between overview and detail. If the program zoom in too much the user will lose overview of the structure of the program, while if it does not zoom in close enough the user will not be able to see the details of the execution of the program. Tanimoto and Glinert (1986) point out that the user can have both if the designers include a miniature map of the entire program in one corner of the screen while the user can still follow the details of program execution in another window.

Cinque et all (1990) propose a methodology based on flow of control diagrams called interface state-transition diagrams (ISTD’s) which would make it possible to separate the visual display from the underlying program and also add another layer of abstraction to the design that allows the designer to concentrate on the important details of the interface. Such a system could be particularly useful for rapid interface design and iconic languages.

5 Graphical representation of data
Another approach to the visualization of a program is to show the flow of data instead of flow of control. Although this representation is most frequently used for debugging purposes, it does not necessarily have to be. Showing the contents of variables can be very valuable for the user, because he or she can get the feel of what’s really happening inside the program.

However, the drawback with having a graphical representation of the program’s data is that it may not be accurate enough. For example, using graph bars to display an integer value may simply not provide the user with enough information. Some things are simply best displayed as simple text.

6 The use of animation
An attractive feature of graphical displays is that you can create animation to better illustrate flow of control or data (Helttula et al, 1990). The term algorithm animation tools are often used to describe such tools and there have been several such animation systems developed over the years.

The usefulness of such animation systems is displayed in its full glory when visualizing sorting algorithms (Duisberg, 1987). By letting the novices actually see elements of the array being swapped they gain a deeper understanding of the different algorithms than by just watching code being executed.

7 Iconic representations
Having looked at the graphical representations of control and data we now move on to the actual elements of such representations, the icons themselves. The design of the icons is a crucial issue and it is tied closely to the metaphor selected and stylization (Tanimoto and Glinert, 1986). Stylization is the combination of visual simplification and exaggeration of important features in such a way that the icon effectively represents the operation in an obvious way (Tanimoto and Glinert, 1986).

Humans acquire information by looking at the spatial relations in complex images faster than by reading a text and it is therefore important to put some effort into the design of icons (Cinque et all, 1990). Furuya et all (1987) propose a general purpose icon system which contains three categories of icons. The first category is the direct icon which is simply a picture of the actual object the icon is supposed to represent. These icons are considered to be intuitive and the user will immediately understand the meaning of the icon. The second category is symbolic icons which uses a more abstract and symbolic representation of the object. Icons in this category usually don’t look like any particular object at all, but it should still be recognizable by using standardization or well known symbols. The last category contains icons that metaphorically represent an object or operation. The icons in this category usually need to be learned since they are not necessarily intuitive. Direct and intuitive icons should be used wherever possible.

Furthermore they divide the contents of an icon into three subtypes: object, operation and merged types. The object icons are the physical objects themselves or in other words the primitives of the visual language on which operations can be performed. The operation icons represent the operations that can be performed on the object icons. Usually metaphor icons need to be used when representing icons of this subtype since operations seldom can be represented by an object directly . Finally the merged icon type has both the elements of the object and operation icons merged into one single icon. These icons are usually created by merging the characteristics of each object and an operation icon.

Dividing the icon contents into such subtypes makes it easier to generate rules on how the different types of icons can look and how they can interact. The Furuya et all (1987) article suggests a set of templates which the icon designers can use to create more uniform icons and most of them are considered to be pretty much the standard in operating systems today such as Windows95 and others.

But even though icon designers today can use templates and standards to help them create good representations, it is important to be aware of the limitations that a graphical display itself imposes to the design of icons, such as resolution, number of colors and so on. Unless the user is using a sophisticated virtual reality to display the space in which the iconic language is visualized, it must be mapped on to a two dimensional screen and that will naturally be restrictive on the language design.

8 HI-VISUAL
HI-VISUAL, an iconic programming language developed by the Hiroshima University, uses icons to represents objects that exist in the real world. There are no icons for functions or procedures. Instead, operations are performed by combining the objects by putting one of the objects on top of another (Kado et all, 1992). By doing so, the designers are able to remove one of the greatest difficulties with visual languages, namely visualizing procedures and functions.

The environment is visualized as an office, in which office-related objects have been placed and the users can interact with the objects in the office in exactly the same way as they would be able to in a real office. A sequence of operations can be recorded by a secretary, and registered as a program and the user can later inform the secretary to run a certain program (Kado et all, 1992).

The system is intended to be used in an business environment and the users should therefore immediately be comfortable with the interface. As mentioned before, since the operations is carried out exactly as in the real world, the user can perform calculations on a spreadsheet by overlapping the calculator icon over the spreadsheet icon, but the user can also wrap the calculator in the spreadsheet paper (Kado et all, 1992). However, the last operation may seem unnatural for certain users if they intended to use the spreadsheet as an input to the calculator and therefore using the spreadsheet icon on the calculator.

Several visual systems like HI-VISUAL has been proposed over the years, but none of them seem to have become widely used. Because, although the interface may be familiar and easy to use, they are simply to limiting and they only seem to be appealing for novices. The user cannot create new objects, only combine the existing objects and use the result from one as input to another object. Visual interfaces like HI-VISUAL are not general purpose languages because they do not offer enough primitives to create entirely new objects, i.e. such as a scheduler if the system does not already offer operations to support scheduler functions.

9 VAMPIRE
Another visual system, VAMPIRE (Visual Metatools for Programming Iconic Environments) developed by Rensselaer Polytechnic Institute, is a tool for rapid generation of iconic programming systems but it is itself implemented as an iconic language (McIntyre, Glinert, 1992). Most visual systems today are implemented using conventional textual languages and VAMPIRE was implemented so that designers could design and implement visual languages in a visual environment. VAMPIRE uses, as any iconic language, program elements (PE) as the iconic language equivalents of textual program statements (McIntyre, Glinert, 1992). The attributes of each icon and how they are laid out in space determine the characteristics of the language and their runtime behavior (McIntyre, Glinert, 1992).

VAMPIRE is implemented in an object oriented way (OO) allowing each element of the language to inherit common attributes from parent objects and classes allowing shared and local attributes to coexist. Three main modules exists within VAMPIRE, the class editor in which the designer can construct a hierarchy describing the structure of the language, the rule and icon editor in which the designer can create the pictorial appearance of the icons and how they can interact and finally the runtime system where the designed iconic language can be run.

The VAMPIRE system was implemented and tested at the Rensselaer Polytechnic Institute and later used as a tool to re-implement already existing iconic languages to monitor the required development time. The results were remarkably good, the iconic languages NOVIS and SUNPICT were implemented using respectively 3 hours instead of 6 months and 15 hours instead of 1 year. McIntyre and Glinert (1992) use these good results as an example on how development time can be dramatically reduced using visual languages.

This comparison is of course not entirely fair. When using VAMPIRE to re-implement the two languages, the designers had all the elements of the language already defined. This was probably not the case in the development with textual programming languages. Still with this in mind the difference in development time is significantly shorter with VAMPIRE.

10 BLOX
A totally different approach, called BLOX, is suggested by Glinert (1986). Simply by translating each textual language construct into an icon you get an unique visual language. The icons are formed as pieces of a jigsaw puzzle and only constructs that form a meaningful series of statements can be combined. The same rules that apply for the textual version also apply for the visual version so basically BLOX is just a visualization of a textual programming language.

This approach would probably feel cumbersome for the experts. The advantages of a visual language, such as reduced development time and better visualization of program structure, is not that apparent in BLOX. The structure of Pascal-BLOX is the same as the textual Pascal language and the iconic version would probably increase development time for the expert user. Selecting the primitive for a WHILE statement and placing it in space will simply take longer than writing WHILE in a text editor.

But a visual language like BLOX would certainly be a valuable tool if used as an introduction to programming languages. The users will learn the syntax of the textual language it represents and they will see which statements fit together. The BLOX language also clearly visualizes the meaning of scope, because only one scope is shown at each level in the visual display.

11 Conclusion
This paper has summarized and discussed some of the obstacles that must be overcome before we will see an increased popularity of visual languages. In fact we see the trend already, the last decade there has been an increased interest in languages such as Visual Basic and Visual C++. Although such languages are not considered to be visual languages, they could be an important step towards an increased popularity of visual languages.

As of yet, we still have a long way to go. There are just too many problems associated with the visual languages that are available today. In short, the most serious problem seems to be that they are either too general or too limited. The general ones try to implement a general purpose programming language and usually end up with an extensive library of iconic primitives in which the user easily can get lost. The limited ones are implemented to solve a specific task and the user quickly finds the language too limiting when trying going beyond the scope of the language. The Tanimoto and Glinert paper (1986) state that iconic languages should be easier to use by people from countries speaking different languages, but they seem to forget that the icons themselves may be a source of confusion. Pictorial representations are by nature ambiguous and what the designer had in mind may be interpreted in a totally different way by the user.

Glinert (1986) believes that we need to find a balance between the ultra sophisticated and the too-simple and that such a balance is the merging of text and graphics in the same language, sort of combining the best of the two worlds. Such a language would allow the user to create each routine in whatever way found most appropriate. People often find redundancy appealing and by combining graphics and text in the same display the user may get the necessary information and power needed to be able to use the visual language as a general purpose programming language.

Many visual languages have been proposed over the years, but few of them have been extensively tested to see if they really work and I think here lies the real challenge of visual languages in the future. The field needs to know what works and what doesn’t. What may work well for one group of people may not work at all for another group and this makes the ladders of learning (Tanimoto, Glinert, 1986) an important element of visual languages. The language needs to grow with the user so that it doesn’t appear too complex or abstract for the novices or too limiting for the more advanced users. Furthermore, many of the visual languages on the marked seem to be very inefficient representations of a programming language and this may be the main cause why visual languages have never quite caught the people’s interest.

References

Chang, S.-K., Costagliola, G., Orefice, S., Polese, G., and Baker, B.R. (1992), A methodology for iconic language design with application to augmentative communication. VL 92, 110-116.

Cinque, L., Costabile, M.F., Levialdi, S., and Missikoff, M. (1990), Towards a formal specification methodology for iconic interface design. VL 90, 149-155.

Duisberg, R. (1987), A gestural interface for animating algorithms. VL 87, 55-66.

Furuya, K., Tayama, S., Kutsuwada, E., and Matsumura, K. (1987), Approach to standardize icons. VL 87 29-38.

Glinert, E.P. (1986), Towards "second generation" interactive, graphical programming environments. VL 86, 61-70.

Helttula, E., Hyrskykari, A., and Raiha, K.-J. (1990), Principles of ALADDIN and other algorithm animation systems. In VLA, 175-187.

Kado, M., Hirakawa, M., and Ichikawa, T. (1992), HI-VISUAL for hierarchical development of large programs. VL 92, 48-55.

McIntyre, D.W. and Glinert, E.P. (1992), Visual tools for generating iconic programming environments. VL 92, 162-169.

Smith, D.N. (1990), The interface construction set. In VLA, 31-51.

Tanimoto, S.L. and Glinert, E.P. (1986), Designing iconic programming systems: Representation and learnability. VL 86, 54-60.