Three researchers at University of Waterloo in Canada have recently developed a unique comic authoring tool that can automatically create stories and comics based on code. This system, called CodeToon, could open interesting possibilities for code-driven storytelling, the process of using computational code to create new stories.

Sangho Suh, one of the authors, came up with the idea for CodeToon and then worked on its development in collaboration with his colleague Jian Zhao, under the supervision of Prof. Edith Law. Their paper, set to be presented in a few weeks at the ACM Symposium on User Interface Software and Technology (UIST), the premier forum for innovations in human-computer interfaces, builds on some of their team’s previous efforts, including the development of a pedagogical tool called Coding Strip and another study leveraging generative conversational AI to teach computational thinking.

“We found that using comics is an effective means for teaching abstract programming concepts, but crafting the comics is tedious and time-consuming,” Jian Zhao, one of the researchers who carried out the study, told TechXplore. “We also identified the opportunities in advanced AI technologies for automating some steps of this task. Thus, we built CodeToon, a comic authoring tool that facilitates this code-driven storytelling process.”

Coding Strip, the pedagogical tool previously created by Sangho Suh and his colleagues at University of Waterloo, was specifically designed to teach programming concepts using comics. CodeToon takes this idea one step further, by automatically creating code-based comics and stories that could be used to train computer science students.

“Inspired by the ladder of abstraction and its relationship to computational thinking, coding strip helps learners move between concrete (comic) and abstract (code) levels of abstraction,” Sangho Suh, one of the researchers who carried out the study, told TechXplore. “Creating coding strips, however, can be time-consuming and laborious, so the primary objective of this project was to make the authoring process more efficient and effortless.”

CodeToon can facilitate code-driven storytelling in two distinct ways. Firstly, it can automatically generate a story template using code. Subsequently, the tool can generate comics based on these code-based story templates.

“CodeToon is an interactive and intelligent comic authoring tool that supports creators to generate stories from code (code→story) and design comics from stories (story→comic),” Zhao and Suh said. “Since comics accurately represent the code semantics and execution process, it enables learners to learn the computational concepts and procedures in an engaging yet effective manner.”

The researchers evaluated the effectiveness of their comic authoring tool in a two-part user study. In the first part, 24 participants with a basic understanding of programming were asked to either use CodeToon or use a series of templates to create story ideas and generate matching comics. Subsequently, all the participants took part in a short interview, where they were asked what worked and what did not work for them.

In the second part of the evaluation, 20 participants who could read and understand Python code were asked to look at 8 coding strips—created using CodeToon or with a series of templates—and share their opinions about how well they mapped to code, illustrated coding concepts, and how useful they thought they might be for teaching.

Based on the interview responses, most participants felt that the 1-to-1 mapping of code onto stories and comics could be an effective teaching strategy, particularly when using comics to alternatively represent code. In addition, those who used CodeToon to generate their comics shared positive feedback about its performance in creating code-based stories and comics.

“Our findings confirm previous literature on multiple representational systems, which recommends making the mapping between multiple representations clear,” Zhao said. “Our insights also shed light on other research areas, such as data and stats comics, which can also leverage this approach to automatically generate comics for learning and teaching concepts in their domains.”

Notably, Suh, Zhao, and Law introduced a new way of looking at code that could have interesting implications. In the future, the same idea could perhaps be applied to other computational and mathematical languages. For instance, future studies could explore the possibility of generating comics and stories based on mathematical equations.

“The most notable achievement of our study, on the surface, may be the efficient process for generating comics from code,” Suh said “But on a deeper level, I think it is the notion of code-driven storytelling. In CodeToon, code drives storytelling: the code’s abstract structure embodying computational ideas such as if-else logic structure becomes the structure of the story and comic. In other words, code determines the sequence and overall plot of the story.”

The tool created by Suh, Zhao, and Law could soon be used in universities and colleges to make learning to code more engaging. In the future, the researchers would thus like to deploy their tool in real classroom environments to better assess its usefulness. In addition, they plan to investigate the possibility of using generative deep learning models, such as GPT-3 and Codex, to fill any gaps in stories and code.

“For example, based on the drawings on the canvas, the system could generate new drawings to add to the existing comic and provide design guidance in real-time,” Zhao explained. “Another possible research direction would be to enhance the authoring process with a more diverse set of sketches and comic resources, more customization and flexibility, as well as the support for more programming languages.”

In his upcoming studies, Suh would also like to investigate the possible use of mathematical languages to drive storytelling. This could ultimately improve the way in which lecturers and students make sense of abstract concepts, complex equations, and procedures.

“There are many exciting research directions we plan to explore in our next studies,” Suh added. “One is to further improve CodeToon, for example, by ensuring that it supports other programming languages and features. Another is to research how we can use CodeToon to integrate art into programming education.”