Mike Leggett
University of Technology, Sydney, Australia

PathScape Prototype: Audio-Visual Indexing in a Landscape

Keywords: landscape, memory, mapping, HUI, GIS

I begin by confessing that I am not a professional art historian but I have curated and reviewed the art associated with the media technologies of the last thirty years, so I do read, I do write and occasionally too, I do think. In so doing I am often motivated to make some kind of visual statement, by making a film, a videotape or a computer-based interactive. The writing associated with this helps prepare procedure and sequence. As Darren Tofts in his book Memory Trade - a Prehistory of Cyberculture has observed: ' … writing is the technology of the inventory'. ¹

A few years ago I wanted to link together some stories about the South Coast of New South Wales. We had spent many holidays there and then one year I bought one of the booklets produced by the local historical society and was amazed to discover that the beach upon which we had spent so many hours was the same beach that Cook had seen for the first time the indigenous people of Australia, members of the Huin nation, one of at least 600 language groups living on the island continent, the Great Southern Land the British then referred to as Terra Nullius.

In 1996 I had just finished work curating an exhibition called Burning the Interface <International artists' CD-ROM> for the Museum of Contemporary Art in Sydney ² and had been exposed to a lot of work - some 130 CDs - which explored the possibilities of interactive multimedia art on disc. This work informed the design of the interactive prototype I subsequently made with a small team of collaborators, about that beach in New South Wales. We sought to present short stories using an audio-visual indexing system and a slightly different approach to screen metaphor.

The traditional Human Computer Interface (HCI) relies heavily on metaphor. The printed page, the desktop, the graph paper, the map, the film soundtrack mix dubbing sheet. These two-dimensional spaces have acquired the bevel edge, two, three and four pixels broad - embossed frames, windows, work areas, palettes, icons, trash cans and so on. The heave off the flat surface of the screen has of course been informed by the Renaissance visual explorers as bold as Mantega or Pinturicchio, drawing the spectator in towards the illusionary spaces that lie inside the back of the video display unit, effected by the contemporary copyist with programming skills, images designed to appeal to the obsessions of electronic gamers pursuing their violent fantasies, just as in the 14th century, we are told, princes, warlords and mercenaries staggered from one bloody encounter to the next.

The sense of presence delivered by the ingenious contemporary games console is extended with three-dimensional real-time rendering, mapping image to wireframe shapes and background perspectives …'As my 16 year-old son moves towards the invader, the rock behind which he prepares his attack still glistens with the blood of a vanquished foe …' Whilst the adolescent craves action and excitement and the rush of adrenalin, how can we begin to divert this technology to become a tool for more complex thought and discourse? Are we moving towards HCIs which complement the workings of human memory and stimulate the individual users imagination?

Random access in the computer as opposed to the book, is capable of circumventing the written word and representing, on demand, the spoken word, sound and picture. Whilst the technical provision for delivering what amounts to little movies on the computer screen has advanced remarkably of late (better compression of files, faster processors and networks), the interface that is an indexing system for sounds and images is yet to arrive. As Derrick de Kerckhove has observed: 'Literate people have a tendency to translate their sensory experience into words and their sensory responses into verbal structures. This comes from the habit of translating strings of printed letters into sensory images in order to make sense of what they read.' ³

Sean Cubitt, in a keynote address to the CHArt Conference in 1998, included a description of ' … post-literacy as the materialisation of the global village.' ⁴ To quote: 'Digital media have drawn upon the formative media of the enlightenment - the history of cataloguing rather than that of books and magazines, and on the history of the map rather than that of perspective depiction - to provide an ordered, navigable and administrable response to the centrifugal forces of electronic networks and user choice. But the history of collections has also a less rational and instrumental side, traceable through the fashion for cabinets des merveilles and Wunderkammers, and forming a curious underbelly to the rationalist carapace of scientific management.' He went on to discuss parallel instances in the past where the browsing metaphor may have existed, not least of all in the field of museology which, through the museum, '…has preserved, as Joseph Cornell's unspeakably evocative boxes show so well, the fascination with selecting intrinsically interesting objects and discovering that side by side, they become more interesting still. That interest, the fruit especially of apparently random orderings, is part of the secret life of the unknown.' ⁵

Fig. 1. Joseph Cornell, Untitled (Cockatoo and Corks) c. 1948, Construction, 14 3/8 x 13 1/2 x 5 5/8 in. Private collection. Sourced from Mark Harden's Artchive.

There is a tension here quite patently between the empiricism of the enlightenment from where so many of our disciplinary patterns derive, and the tendencies within popular culture and post-modern interdisciplinary studies. The notion of visual indexing traditionally is based upon word interpretation - keywords associated in online picture libraries with location, subject, colour, date etc.

Academic sites, such as those collected on the CHArt site, http://www.chart.ac.uk/vlib/images.html, range from the Visual Art Data Service (VADS) image resources site to the superb Exeter Cathedral site, http://www.exetercathedral.tell-com.com/, which links treatise, catalogue and map to images using the hypertext book model. 'There are two main routes into the material, Visual and Verbal. […] The Verbal route is for those who are more at ease with text than images.' There is an elegance and appropriateness in this site that links a contemporary database design with a medieval equivalent that also sets out to store knowledge using systematic method. ⁶

Other websites have experimented with delivering knowledge via a combination of archive film footage, slides and text using intelligent interfaces that for instance, monitor your selection and then reorganize subsequent options to cluster related topics together with a combination of image and words re-shuffling their indexical position. One of the earliest ones was developed by a research team in the Media Lab at MIT. Jerome B. Wiesner, 1915-1994: A Random Walk through the 20th Century, http://ic.media.mit.edu/projects/JBW/. ⁷

As a pastime, clicking the mouse from one image or word to the next, surfing from one server to another is hardly of concern to the highly motivated user. However, the focus required to make the Internet session or exploration of any interactive artefact productive, requires quite prodigious use of personal memory and the ability to store and retrieve whatever it is, text, images, sounds - data. Bookmarks, electronic as with real books, are prone to disappear and anyway, once you are past a dozen or so, without an indexical notation how are they retrieved? Likewise, entering a new building, navigating around a new town or city, memory is tested. We use maps and street names at first but essentially we are learning the visual coordinates - the image of the flower-shop, the park with the fountain, the shape of a skyline, the pub on the corner. Eventually we have recorded the route in memory enabling us to play the movie later in order to arrive at a pre-determined destination.⁸

There are software tools related to topography, recorded time and place, and these are widely used in the industries related to so-called environmental planning - water and land management, urban layout, national parks, mining and agriculture, etc. These are ingenious, specialised tool sets based on data derived from that old trusty of the Enlightenment and scientific method, measurement. Combined with GIS satellite data and a range of plug-ins that enable digital images, sound and text files to be attached to specific coordinates, this allows extensive profiles to be constructed and navigated in real-time.⁹ One of the examples is the University of Sydney Timemap Project, http://www.timemap.net.

The Humanities have adapted these tools - archaeologists and social scientists most notably. In the west of Sydney, the City of Fairfield has commissioned a website using the widely used ArcView application linking through a combination of text and the map metaphor, personal oral histories with localities. The City of Fairfield oral history project is at http://www.acl.arts.usyd.edu.au/fairfield/

The time taken to memorise a route, for real or via the HCI, varies from person to person. It could be the linguist de Kerckhove is right when he observes: 'When Tiresias tells Oedipus that he is blind to the truth of his position, what he is really implying is that Oedipus' excessive reliance upon his eyes and visual logic … ' - to alphabetic literacy - ' … has made him blind - or, more accurately deaf - to anything beyond the evidence. The fact is that, as western people, we have become gradually deaf through no fault of our own, through the rewiring of our nervous system by literacy.' ¹⁰

The Greeks orators and rhetoricians, before the alphabet had been handed down, had developed an elaborate form of artificial memory, ars memoria, described in Frances Yates book from 1966, The Art of Memory. Ars memoria essentially imprinted on the memory '… a series of loci or places. The commonest, though not the only type of mnemonic place system was the architectural type. […] We have to think of the ancient orator as moving in imagination through his memory building whilst he is making his speech, drawing from the memorised places the images he has [previously] placed on them. The method ensures that the points are remembered in the right order, since the order is fixed by the sequence or places in the building.' ¹¹

Thus the term 'classic film narrative' adopts another meaning. The first movies were a conceptual model made by the Greek rhetoricians, complete with wide shots, tracking shots, panning, tilts, close-ups and flashbacks, all played in the cinema of the mind's eye. The story, the diegesis of cinema, was equally inexorable from beginning, through middle to end. In this, the age of Deleuze and Guattari's rhizome, linearity need not structure thought within the confines of logic and rhetoric.¹² In the same way as the walk from home to the station may allow interventions of the everyday to structure the day itself, even enhanced by the imprecision of the visual cues that guide us during the walk, then too the invention or re-invention of a visual literacy based on the newer technologies, would enable us (with the happenstance of chance encounter), to employ indexing and classification appropriate to the task in hand, not subject to the reductive tendencies of the inventory and the catalogue that are liable not to stimulate but to stifle imagination.

The interactive multimedia prototype of PathScape I developed has an interface and navigation system which gives access to knowledge through a connection with a specific place or location. It seeks to enable the navigator to associate digital documents with a (fragmented) representation of contiguous cinematic space and thereby offer a means of retrieval based on visual memory, as the prototype demonstrates.

Fig. 2. Screensaver image from PathScape prototype. © Mike Leggett

There are thirty stories in this prototype. They are all related to this part of the South Coast of New South Wales in some way, and I have selected for this occasion two of the stories which connect the work of visual artists with the storage, preservation of knowledge combining time and place. It is ordered sequentially by the index device of ecological zones, signalled by changes in background colour and background sound, the first being the beach, which we can scan in 360° in either direction and within which we can find the knowledge we are seeking.¹³

Fig. 3. The sailors of the L'Astrolabe, share their catch of fish with the natives, lithograph based on the sketches of Louis Auguste de Sainson. National Library of Australia, Canberra.

The sailors of the L'Astrolabe, share their catch of fish with the natives is a lithograph based on the sketches of Louis Auguste de Sainson, artist on the French corvette in 1826, one of several scientific expeditions the French sent to the Great Southern Land in the 18th and 19th centuries.¹⁴

Fig. 4. At the completion of the narrative, colour coded buttons appear: Green for contextual history; Blue for image analysis; Black for accessing the text-based index; Red moves back to the previous level. © Mike Leggett

The circles that appear at the end of the narrative sequence branch to the associated topics. The programming framework is designed to be dynamic and enable media assets (in this prototype, movies, graphics and sound), as new discrete content stories can be added and recognised each time the application is launched. Following demonstrations to several groups of the first prototype, another feature was added for those who had difficulty in retrieving stories lodged in places represented on 'the walk'. The local aboriginal land council too wanted to know precisely who it was speaking, whose work was being looked at and where the places were in 'their country' - specificity was their need. The grey/black circles on the screen that sit behind each of the coloured circles are the control factor in this second prototype (the cop-out if you like from the original intention), being the route through to the traditional text-based navigational interface. From here, in the way with which we are familiar, we are able to launch according to the encoding of the word literate technologies. The user in this prototype therefore has a choice - to search the index by using images and sounds, or to search using words.

Fig. 5. Ulladulah Mickey Peterbrough Steamer by Mickey or Willy the Cripple from Ulludulla. Pen, ink, crayon and pastel on surveyors paper, c. 1888. AIATSIS Collection, Canberra.¹⁵

One of the other interactive features in this interface is the ability of the user to scan across the detail of a high-resolution image whilst the commentary runs. Giving the user something to do whilst listening to what is essentially a linear lecture, enabling them to shuffle as it were the picture experience, I feel generally enhances the engagement with the subject. The PathScape framework can be delivered on disc (CD or DVD) or via the Internet or broadband cable or conceivably, as it uses XML protocols, via a PDA or mobile phone.

The image and much of the sound components of the prototype, are delivered essentially as a series of interrelated QuickTime movies. Other tools we considered for delivering images included QuickTime VR, whereby a scan of shots can be 'stitched' together and then set up with 'hot spots' that enable the user to move from location to location, slide to slide, whilst the lecturer's voice is heard. Does this turn into a battle for attention? Are you listening? Or are you watching?

My final example indicates ways in which the storage and retrieval of non-text based information may be further enriched in the future. Dr Stephen Barrass of the CSIRO, the Australian science and industry federal research organisation, recently demonstrated the work his team is pursuing with haptic work benches. Looping back again to Capt James Cook's, and his snuff-box, which turns out to be an extraordinarily valuable artefact to Australian neo-colonialists and the National Estate, far too valuable to be touched by mere mortals. Barrass has rendered the snuff-box into a computer installation which enables the researcher using a force feedback arm, called phantom, to turn the box over and around, to feel and hear the embossed surface and to essentially experience it as if the real object, though it remains simply a data set interpreted by some software into a series of visual cues for interaction (see http://www.cmis.csiro.au/imvs/immerse).

Conclusion

Greg Ulmer's work around the term chorography is useful here. Based, again, on one of Plato's metaphors, the chora in the Timaeus, ' … portrayed as the generative space that intervenes between being and becoming …'. As Darren Tofts summarised: 'Chorography is to hypermedia what the art of memory was to the oral tradition. It sets the scene for imagining 'an electronic way of reading, writing and reasoning' that, while grounded in the concepts of memory and place, recognises that these concepts are subject to modification. … The new world of immersion in information, as opposed to retrieval of information, is the frontier that chorography sets out to map.' ¹⁶

When reviewing Memory Trade three years ago I suggested: 'We do not steer a collision course or engage with arguments in cyberspace, as the 17th-century apparatus of the book encouraged us to, but rather enter a state of consciousness which through realisation leads to states of resolution. Reason by contrast, proposes a continual forward movement (such as the growth of capital), that challenges the finite - resources, time, morality etc. - through the creation of ever increasing numbers of goods, for a decreasing population prepared or able to exchange and trade.' ¹⁷ The continual movement forward (referred to as 'growth' and 'progress' in the culture of Capital) is encountered in various narrative forms, where linear development of the subject subsumes the reader or the viewer's ability to intercede. Interaction with the rhizomatic development of related topics, using visual and audible cues as in the example of PathScape, (besides closing the gap within electronic networks between the storing and retrieving knowledge), propose an approach to indexing which more closely complements the working of human memory whilst stimulating the individual user's imagination.

I'll end on an aphorism from McLuhan's assistant, de Kerckhove as he directs ' … us away from the prescriptive of the literate ear and toward the associative of the oral ear … ', the overall diminution of our sensory apparatus, even Oedipus' excessive reliance upon his eyes and the visual logic of the alphabet: 'Our neglect of the ear may be one of the prices we have paid for literacy'.¹⁸

November 2002

Notes

1. Tofts, D. (1997), Memory Trade – a Pre-history of Cyberculture, Interface: Fine Arts Press, Sydney and G+B Arts International.

2. Leggett, M. & Michael, L. (1996), Burning the Interface <International artists' CD-ROM> Catalogue, Museum of Contemporary Art, Sydney.

3. De Kerckhove, D. (1995), The Skin of Culture, Somerville, Toronto, p. 81.

4. Cubitt, S. (1999), 'Catalogues, Charts and Cabinets: Problems of Order in Multimedia', Computing and Visual Culture. Representation and Interpretation. CHArt Conference Proceedings, vol. 1, London, p.13.

5. Cubitt, S., op.cit., p. 12 and 21.

6. I was in Exeter at the time this visual material was being collected by Avril Henry and Anna Hulbert and was able to climb the scaffold and shoot some film of these extraordinary medieval constructions which has a lot to do with the interdisciplinary project of the Church of the time, incorporating popular culture within its disciplinary structures and thus extending the mythologies so peculiar to the Church of England.

7. Glorianna Davenport, Cheryl Morse, Michael Murtaugh, Freedom Baird, Richard Lachman, Peter Cho, Phillip Tiongson, Laughton Stanley Research Team. Michael Murtaugh, Interface Designer. 'This hyper-portrait introduces the audience to a remarkable man whose life centered on science, government, education and issues of cultural humanism. Early in his career, Jerome Wiesner developed an audio recording laboratory at the Library of Congress and traveled extensively throughout America, capturing folk music by native performers. He directed MIT's Research Lab for Electronics during the Cold War, served as National Science Advisor to John F. Kennedy, and eventually became President of MIT. After the end of World War II, Wiesner became a prominent advocate of disarmament and was a key player in negotiating the first Nuclear Test Ban Treaty. In this hyper portrait (which runs on the World Wide Web), we invite viewers to explore the Twentieth Century through an extensible collection of stories about and recollections by the central figure. We also invite viewers who knew JBW to share a memorable story with our growing society of audience. Jerome B. Wiesner uses the Dexter continuity engine developed by Michael Murtaugh. This work is delivered using two principle channels: the World Wide Web and a CD-ROM.' http://ic.media.mit.edu/projects/JBW/

8. Or not arrive. For those of you who have been to such places as our federal capital Canberra, playing back this journey movie in memory runs rather like the film Memento: where everything looks the same, where short-term OR long-term memory doesn't help, with or without the use of a Polaroid camera, and where time goes both backwards and forwards in linking loops … Memento (2000) screenplay and director Christopher Nolan.

9. 'Taking care of business means leveraging technology that helps you see the big picture, make the best decisions, and capitalize on your organization's investment in data and resources. ArcView® GIS software by ESRI offers an innovative solution that will help you create, visualize, analyze, and present information better and more effectively. ArcView GIS software is the premier solution for desktop GIS analysis and map presentation. With ArcView GIS you can create intelligent, dynamic maps using data from virtually any source and across most popular computing platforms. ArcView GIS includes tools and data you can use immediately to perform state-of-the-art analysis on key issues. It lets you work with maps, database tables, and business charts all in a single application.' ArcView GIS electronic publicity pamphlet (2000).

10. De Kerckhove, D. (1995), The Skin of Culture, Somerville, Toronto, p. 105.

11. Yates, F. A. (1992) The Art of Memory, London: Pimlico (1st ed. 1966), p. 18f. The discipline also emphasises the use of 'unusual' images to stimulate the mind in pursuit of the aide-memoire. The highly unusual images constructed by the stonemasons of Exeter Cathedral, who incorporated local myths and superstitions into the bosses high in the vaulting, may be a medieval manifestation of the same principle.

12. Deleuze, G. & Guattari. F. (1994), A Thousand Plateaus: Capitalism and Schizophrenia, trans. B. Massumi, Minneapolis: Univesity of Minnesota Press.

13. 'Les marins de L'Astrolabe partagent leur peche avec les Naturels', lithograph by Victor-Jean Adam based on the sketches of Louis Auguste de Sainson, artist on the corvette L'Astrolabe, 1826. Words in the CD-ROM are by Mike Leggett and based on the Journal of Capitain Dumont D'Urville and the notes of Michael Organ in 'Illawara & South Coast Aborigines 1770–1850', (1990), Aboriginal Education Unit, University of Wollongong, NSW, Australia. Transcript: 'This lithograph of the sharing the fish between the crew of the Astrolabe and the aborigines was made in 1826 at a time when it was usual for an artist or lithographer to partially cloth naked bodies in the name of decorum, even though reality was otherwise. However, in this print the natives are portrayed as they were, naked except for some instances of small girdles wrapped around their abdomens and the wearing of European jackets. It is known from other sources that the Aborigines were usually scantily clad but that they also employed possum rugs for warmth during winter and at night. As the encounter with the French occurred in late November, the southern early summer period, it is likely the natives were clothed as depicted in the lithograph.

We should also remember that just as it took European artists many years to accurately draw unfamiliar Australian animals such as the kangaroo, koala and platypus, so too did de Sainson experience difficulties in correctly portraying the features of the Australian aboriginal. Like many other artists of the time, he gave the aboriginal, European facial features.'

14. In the French title to this lithograph the term 'naturels' is used – besides translating as 'native' the word in Cassell's French dictionary (1920) can also mean: natural, innate, artless, plain, homebred, genuine, unsophisticated. This to me indicates some of the problems associated with keyword searches.

15. 'Peterbrough Steamer' by Mickey or Willy the Cripple from Ulludulla, pen, ink, crayon and pastel on surveyors paper, c.1888. Slide AIATSIS Library. Text by Mike Leggett. Transcript from CD-ROM: 'This picture was made by an artist known as Ulladulla Mickey, or Willy the Cripple from Ulladulla. The artist is the earliest known south coast aboriginal to employ pen, ink, crayon, and pencil on paper using European techniques. This image entitled The Peterborough Steamer is thought to have been made in 1888 when Mickey was over 60 years of age. Throughout his life he would have witnessed the gradual appropriation of his traditional lands by the English. The sea was the most effective means of reaching the lands, capable of producing returns in timber, wool, fish and precious metals and the most efficient means of removing these gains back up the coast to Sydney Town – the technology of shipping and ships was essential to achieve these ends. The picture displays the range of shipping technology that Willy witnessed during this period. The creatures of the sea and the bush however, are also present. These totemic creatures, many of whom would be closely related to the indigenous people, use a means of representation that was within the aboriginal tradition. Flat rocks up and down the coast would be inscribed with creatures similar in appearance and depiction. The statement made is clear. Whilst all manner of shipping may ply up and down the coast carrying figures dressed in all their finery, the creatures of the Bush and of the Sea, quite literally surround the incursors – to top, to bottom, to North and to South.

16. Tofts, D. (1997), Memory Trade – a Pre-history of Cyberculture, Interface: Fine Arts Press, Sydney and G+B Arts International, p. 72f.

17. Leggett, M. (1999) Memory Trade, Leonardo Digital Reviews, MIT. http://mitpress.mit.edu/e-journals/Leonardo/ldr.html

18. Quoted after Leggett, M. (1996) Literate Ears, RealTime /11, Open City, Sydney.