What is multimodality anyway?
Arola, Sheppard, and Ball colloquialize multimodality in their book Writer/Designer as the combined use of visuals, words, spatial relations, gestures, and sound to communicate. You run into multimodal communication every day! If you’re watching a TV commercial, scrolling through Twitter, or browsing the interwebs, multiple modes of communication are tag teaming to seize your attention.
Give me an example.
Ok, let’s run with the first example – a TV commercial. Watch below or follow this link to watch NSYNC’s rendition on a classic Chili’s commercial.
The video hits home the boys’ love for ribs using several modes. For example, pause the video at 0:03 and notice a young, Ramen-haired Justin Timberlake drawing roman numerals on a rock. Each of those marks symbolizes a day Justin has been without his Babyback Ribs. These textual signals are layered in the video to emphasize his desire for ribs.
Aural cues are also given to make the commercial seamless. There’s likely not a helicopter in the shot with NSYNC, but helicopter noises are integrated throughout to make it sound like one is. Even the song they’re singing adds another stratum of sound. Each works in tandem to further drive home how delicious Chili’s’ ribs are. There are many other examples of multimodality here, but let’s move on to another example.
If you’ve ever been on YouTube, you’ve likely watched a Fine Brothers Entertainment production before. If not, here’s your chance. Watch below, or follow this link to see Adults React to the Macarena.
Immediately upon loading this video, your eyes are assaulted with spinning visuals and loud, yellow pop-up text. There are two modes right there – visual and textual. Text also accompanies each person, telling us who they are. What they’re watching is also on the screen. So are fun facts and YouTube comments. These modes are overwhelming but keep you interested and together give a comprehensive view of the topic.
Another mode at work is body language. Each reactor expresses their enjoyment, surprise, and nostalgia with their face. Go to 3:52. Emily is my favorite to watch because she uses her hands to emphasize everything she says, and it makes her easier to connect with.
Music, also, creates a mood throughout, proving that communication doesn’t only use your eyes. But, just like those modes, it’s dictated by context. You wouldn’t play AC/DC at a funeral (or, maybe you would, I don’t know you). Music is a deliberate, man-made force, so don’t underestimate its potency. Let’s look at one last example.
One last example.
Even if you aren’t experiencing a full-blown, commercial production, multimodality can be at work. Look at this tweet from J.K. Rowling.
The emoji hands are speaking to you, letting you know Queen J.K. approves of @morganhousel’s tweet. Again, text and pictures team up to communicate.
Even the organization of this tweet works. By boxing and indenting @morganhousel’s tweet, the most casual Twitter user will know it’s the object of her reaction. These kinds of spatial relations work subtly to shape our experience.
So, see. Multimodality is all around. Hardly anything we see is text or video or audio alone. Instead, each works simultaneously to create a fuller experience, compensating for another mode’s strength or weakness. Communicators – beware. Using multiple modes of communication is the best way to reach your audience. It will make your work more relatable, enjoyable, and effective.