Overview
Last week, we learned about how the shift to digitality (the shift, as Nicholas Negroponte calls it, from “atoms to bits”) causes different kinds of data to converge into texts called multimedia.
This week, we are going to discuss a term for this convergence called “multimodality” which speaks specifically to the possibilities of the digital age specifically as opportunities for new kinds of compositions
What is a mode?
The semioticians Jeff Bezemer and Gunther Kress describe a “mode” as:
a socially and culturally shaped resource for making meaning. Image, writing, layout, speech, moving images are examples of different modes.
Modes are different ways of communicating, and messages can be conveyed in a variety of these modes. Moreover, as Bezemer and Kress clarify, a communicative mode has particular modal resources available to it. For instance, writing has grammar, syntax, and word-choice available to it. So does speaking. However, writing also has a variety of graphical resources, such as font face, size, punctuation and even color in certain cases. Speech has none of those, but it does have intonation and variation in loudness. Anyone who has ever tried to text someone sarcastically knows intuitively about modal resources.
Thus, we become conditioned, as we move through our communicative world, to certain modes having certain affordances, such that writing and speaking are, for instance, seen as having different comparative strengths and thus different uses for certain circumstances.
Media
Mode is often considered alongside medium, which is the material form in which a message arrives. We talk about writing as a mode or speaking as a mode, but this belies the fact that writing can be via email or via a hand-written letter; speaking can be a recorded TEDTalk on YouTube or chatting with a friend after class.
Marshall McLuhan, the founder of the discipline known as media studies, once famously claimed that, as far as communication was concerned, “the medium is the message,” in that how a message arrives is more important than the content of that message. McLuhan, who himself was a bit of a showman, was famous for making pithy and absolute summations of his ideas, and, as people have explored in more detail, modality is increasingly an important component of media: modes are the techniques through which a medium reshapes its message.
Multimedia, Multimodality
As the digital convergence of media we saw in Negroponte develops, we see more media coming to resemble one another. For instance, a FaceTime video call on my iPhone blends elements of television with elements of the telephone in ways that would have been incomprehensible to earlier generations.
The most important convergence, for our purposes in this class, is the way in which printed text becomes one mode amongst many. Previously, writing was usually encountered as a wall of text, organized into rows and columns, on a printed page, whether in a newspaper or a book. If there were illustrations, they were static photographs of low quality (in a paperback book or a newspaper) or higher quality in magazines. The ability to interact with an image or a text was nil: you can’t zoom on a magazine, you can’t search in a paperback book (as much as I want to).
But on a medium such as the Internet, the digital nature of all media allows for these very things: I can search for words in a document, charts illustrating my point are interactive and can be experimented upon by the user. Multiple modes are available for my consideration when I write. This is both a blessing and a curse.
On the one hand, these new multimodal texts offer more affordances to connect to the reader but, at the same time, more resources have to be managed by the author. This is your task as a budding digital author.
5 Modes of a Digital Text
To help get you started thinking about the composition of multimodal texts, consider five major modes that can be engaged by a digital text:
- Linguistic
- Spatial
- Visual
- Aural
- Gestural
The linguistic mode concerns the things must usually associated with writing in English classes: word choice, sentence structure, paragraphing, and punctuation. While these resources are now amongst many others, we still have to think carefully about how our choices in terms of grammar and words impact our perception as writers: are we constructing ourselves as having authorities? Are we engaging our readers in a way they find pleasurable? Does our text produce the right kind of engagement?
The spatial mode concerns arrangement of the screen. This can refer to physical arrangement in three dimensions (left and right, top and bottom, and even front and back), but it can also refer to correctly creating an information hierarchy that uses increasing and diminishing text size and color to indicate important and less important information. Even image placement can be considered under the spatial mode.
The visual mode concerns the appearance of the text in terms of color and appearance (but remember that spatial modes also provide a visual shape to the text; these modes often blur). Considerations here might cover charts and videos, but also would concern things like color choices and the choice and size of fonts.
The aural mode concerns how we listen to a text. This might cover things like music, but it can also cover the intonation, inflection, and accent used in narration or interviews that shape a reader’s experience of a text.
The gestural mode concerns how the eye is drawn through a text by movement. In public speaking, this is often accomplished through gesture. However, even texts less focused on video or speaking have to concern themselves with gesture. For instance, animations are often used to imply relationship between components in a text (Google’s Material Design specification has a fascinating account of how motion is utilized). Similarly, designers often speak of “implied direction,” the idea that humans are primed to look at images and determine which way they point. We think cars point toward their front and arrows point toward their narrow end. An awareness of this can help imply purpose and navigation in a digital text.
How To Use This?
Multimodality implies two related approaches. On the one hand, I can consider how to use the modes in my own composing. On the other, I can ask myself how a text is or is not effectively utilizing the modes in appealing to me as a reader.
For your lab this week, you will be utilizing the five modes of digital texts to consider an existing web text.