ARTIFICIAL INTELLIGENCE (AI) ASSISTED MULTIMODALITY IN DIGITAL CAMPAIGN TEXTS IN THE PRESIDENTIAL ELECTION

Dian Marisha Putri; Mulyadi Mulyadi; Rahmadsyah Rangkuti; Amrin Saragih; Yusni Khairul Amri

doi:10.22437/jiituj.v10i2.42675

Authors

Dian Marisha Putri Universitas Sumatera Utara https://orcid.org/0000-0001-8126-4611
Mulyadi Universitas Sumatera Utara https://orcid.org/0000-0001-8930-1325
Rahmadsyah Universitas Sumatera Utara https://orcid.org/0000-0003-4419-8648
Amrin Saragih Universitas Negeri Medan https://orcid.org/0000-0002-3914-8569
Yusni Khairul Universitas Muhammadiyah Sumatera Utara https://orcid.org/0000-0002-2934-0696

DOI:

https://doi.org/10.22437/jiituj.v10i2.42675

Keywords:

Campaign Texts, Election 2024, Multimodality, The Role

Abstract

This study examines the multimodality of verbal and visual modes in 2024 digital campaign texts combining linguistic approaches with artificial intelligence technology. Verbal mode analysis involves the use of multiple modes in discourse (Kress & Van Leeuwen, 2001). The use of AI helps identify patterns of visual-verbal relationships. The data and data sources are campaign texts for the 2024 General Election. Five social media platforms were used: WhatsApp, Facebook, Twitter, TikTok, and Instagram. Five online media platforms were selected: a) Detik.com, b) Kompas.com, c) Tribunnews.com, d) Liputan6.com, and e) cnbcindonesia.com. A total of 200 election texts were collected between May 2023 and February 2024. The collected data were qualitatively reviewed in the form of digital texts. The analysis results illustrate that AI-assisted multimodality in explaining the meaning of text in visual and verbal modes, such as: a) strengthening the meaning of the text. b) using a sarcastic style of language. c) The role of meaningful visual text has a relational nature: between illustrations and silhouettes, followed by the attributes of text as a material process. d) representing campaign text refers to how the semiotic system on objects and relationships outside the text and context, both directly and indirectly. The findings demonstrate that the visual language representing political campaign text serves as both an action and a reaction process, as well as a verbal and mental process, within the context of the situation.

Downloads

Download data is not yet available.

Author Biographies

Mulyadi, Universitas Sumatera Utara

Dept. of Indonesia Lit

Rahmadsyah, Universitas Sumatera Utara

English Dept.

Amrin Saragih, Universitas Negeri Medan

English Language Study Program

Yusni Khairul, Universitas Muhammadiyah Sumatera Utara

Indonesian Language Education Study Program

References

Adami, E., & Kress, G. (2014). The social semiotics of contemporary communication. Visual Communication, 13(3), 1-14.

Austin, J. L. (1962). How to do things with words. Oxford University Press.

Bateman, J. A. (2014). Text and image: A critical introduction to the visual/verbal divide. Routledge.

Bezemer, J., & Kress, G. (2016). Multimodality, learning and communication: A social semiotic frame. Routledge.

Bocheara, A. (2015). The role of religion in shaping politeness in Moroccan Arabic. Journal of Politeness Research, 11(1), 71-98.

Bolognesi, M., & Lievers, F. S. (2020). How language and image construct synaesthetic metaphors in print advertising. Visual Communication, 19(4), 431-457.

Boulianne, S. (2019). Social media use and political participation: A meta-analysis. New Media & Society, 21(8), 1871-1891.

Bucur, C. (2020). Detecting multimodal fake news: A survey. ACM Computing Surveys, 53(5), 1-38.

Budiardjo, M. (2015). Basics of political science (Rev. ed.). Gramedia Pustaka Utama.

Chaer, A. (2007). Language studies: Internal structure, usage, and learning. Rineka Cipta.

Chen, Y., & Zhang, Y. (2021). Sarcasm detection in social media: A survey. IEEE Transactions on Affective Computing, 12(4), 876-893.

Chen, Z. T., & Cheung, M. (2022). Consumption as extended carnival on Tmall in contemporary China. Social Semiotics, 32(2), 163-183.

Dos Santos, T. A., & Lopes, I. F. (2025). Is Social Media a Democratic Forum for Public Accountability in Times of Crisis? The Brazilian Government's Response to the COVID‐19 Pandemic. Abacus, 61(1), 194-217. https://doi.org/10.1111/abac.12356.

Eggins, S. (2004). An introduction to systemic functional linguistics (2nd ed.). Continuum.

Enli, G. (2017). Twitter as arena for the authentic outsider. European Journal of Communication, 32(1), 50-61.

Entman, R. M. (1993). Framing: Toward clarification of a fractured paradigm. Journal of Communication, 43(4), 51-58.

Fazio, X., Gallagher, T. L., & DeKlerk, C. (2022). Exploring adolescents' critical reading of socioscientific topics using multimodality texts. International Journal of Science and Mathematics Education, 20, 93-116.

Finch, G. (2000). Key concepts in language and linguistics. Palgrave Macmillan.

Gazdar, G. (1979). Pragmatics: Implicature, presupposition and logical form. Academic Press.

Giachanou, A., & Crestani, F. (2022). Multimodal fake news detection: A survey. Information Processing & Management, 59(4), 102-118.

Graakjær, N. J. (2019). Sounding out "I'm lovin' it"—A multimodality discourse analysis. Critical Discourse Studies, 16(5), 569-582.

Grabe, M. E., & Bucy, E. P. (2019). Image bite politics: News and the visual framing of elections. Oxford University Press.

Hangloo, S., & Arora, B. (2022). Combating multimodality fake news on social media. Multimedia Systems, 28(6).

Habal, M. B. (2023). Creative Global Communications: A Game Changer in the New Digital Information Technology Arena—Still Competition Between Structure and Function. Journal of Craniofacial Surgery, 34(4), 1157-1158. https://doi.org/10.1097/SCS.0000000000009447.

Hou, S., Zhou, S., Liu, W., & Zheng, Y. (2018). Classifying advertising videos by topicalising high-level semantic concepts. Multimedia Tools and Applications, 77(19), 25475-25511.

Jewitt, C. (2016). Introducing multimodality. Routledge.

Jewitt, C., Bezemer, J., & O'Halloran, K. L. (2016). Introducing multimodality. Routledge.

Jin, Z., Cao, J., Guo, H., Zhang, Y., & Luo, J. (2017). Multimodal fusion with recurrent neural networks for rumor detection on microblogs. Proceedings of the 25th ACM International Conference on Multimedia, 795-803.

Johnstone, B., & Marcellino, W. M. (2010). Dell Hymes and the ethnography of communication. In The SAGE handbook of sociolinguistics.

Karim, M. A., & Erwhintiana, I. (2020). Perlokusi speech patterns in the web series Behind the Heart. Pujangga Journal, 6(2).

Kelly, L. B., & Kachorsky, D. (2022). Text complexity and picturebooks: Learning from multimodality analysis. Reading and Writing Quarterly, 38(1), 33-50.

Khattar, D., Goud, J. S., Gupta, M., & Varma, V. (2019). MVAE: Multimodal variational autoencoder for fake news detection. The World Wide Web Conference, 2915-2921.

Kramsch, C. (2002). Language and culture: A social semiotic perspective. ADFL Bulletin, 33(2), 8-15.

Kreiss, D., Lawrence, R. G., & McGregor, S. C. (2018). In their own words: Political practitioner accounts of digital media use. Political Communication, 35(2), 179-198.

Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. Routledge.

Kress, G., & van Leeuwen, T. (1996/2006). Reading images: The grammar of visual design. Routledge.

Kress, G., & van Leeuwen, T. (2001). The modes and media of contemporary communications. Arnold.

Kusmanto, H., & Widodo, P. (2022). Positive politeness strategies during online learning. Studies in English Language and Education, 9(3), 1170-1182.

Kusno, A., Arifin, M. B., & Mulawarman, W. G. (2022). Identifying the virtual extralingual context of social media language. Diglosia, 5(1S), 261-282.

LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.

Lim, M. (2020). The algorithmic management of polarization and democracy in Southeast Asia. In Digital Disinformation in Southeast Asia. ISEAS Publishing.

Macagno, F., & Pinto, R. B. W. S. (2021). Reconstructing multimodality of arguments in advertisements. Argumentation, 35(1), 141-176.

Machin, D., & Mayr, A. (2012). How to do critical discourse analysis: A multimodal introduction. Sage Publications.

Mahsun, M. (2005). Language research methods. Raja Grafindo Persada.

Martin, J. R., & Rose, D. (2007). Working with discourse: Meaning beyond the clause (2nd ed.). Continuum.

Matthiessen, C. M. I. M. (2014). Halliday's introduction to functional grammar (4th ed.). Routledge.

Messaris, P., & Abraham, L. (2001). The role of images in framing news stories. In S. D. Reese et al. (Eds.), Framing public life (pp. 215-226). Routledge.

Miles, M. B., & Huberman, A. M. (2014). Qualitative data analysis: A methods sourcebook (3rd ed.). Sage Publications.

O'Halloran, K. L., Tan, S., & Marissa, K. L. E. (2017). Multimodal analysis for critical thinking. Learning, Media and Technology, 42(2), 147-160.

Robbins, S. P. (2008). Organisational behaviour (B. Molan, Trans.). Intan Sejati.

Rochmawati. (2020). Pragmatic and rhetorical strategies in English-written jokes. Pujangga Journal, 6(2).

Rossini, P. (2022). Beyond incivility: Understanding patterns of uncivil and intolerant discourse. Communication Research, 49(3), 339-363.

Rustono. (1999). Principles of pragmatics. IKIP Semarang Press.

Salsabila, N., et al. (2021). Perlocutionary speech acts in the dialogue of Imperfect film. Allegory Journal, 1(2).

Schiffrin, D. (2007). An introduction to discourse studies. Student Library.

Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, H. (2017). Fake news detection on social media: A data mining perspective. ACM SIGKDD Explorations Newsletter, 19(1), 22-36.

Van Leeuwen, T. (2015). Multimodality. In D. Tannen et al. (Eds.), The handbook of discourse analysis (2nd ed., pp. 447-465). Wiley Blackwell.

Vosoughi, S., Roy, D., & Aral, S. (2018). The spread of true and false news online. Science, 359(6380), 1146-1151.

Wang, Y., Ma, F., Jin, Z., Yuan, Y., Xun, G., Jha, K., & Gao, J. (2018). EANN: Event adversarial neural networks for multimodal fake news detection. Proceedings of the 24th ACM SIGKDD International Conference, 849-857.