A humanist take on scientific computing

Finnish post-doctoral researcher Tuomo Hiippala is part of a new generation of humanist researchers, using powerful computational resources to boost their research.

Combining a PhD in English Philology with a keen interest in computer vision and machine learning Hiippala is developing new ways of handling large collections of images.

“Roughly 30 years ago, computers transformed the empirical study of language,” Tuomo Hiippala explains.

“Optical character recognition, an established subfield of computer vision, enabled the creation of massive collections of words (corpora) to be studied. Now, given the advances in describing images, I think computer vision can be leveraged again to understand more visual forms of communication.

“I’m working on using computer vision and machine learning to handle large collections of images, and to group them together based on similarity, in order to help the researchers working on visual communication to cope with increasingly large amounts of visual data available in social media and historical archives”.

Studying multimodality

Tuomo Hiippala specializes in studying the interaction between multiple forms of communication, as in the interaction of language, image and graphic design in for instance magazines, newspapers, and many other printed or digital texts.

This phenomenon is generally termed “multimodality”, and Hiippala has published a book on conducting empirical research on multimodality, based on his doctoral dissertation, for which he created a corpus of tourist brochures published by the city of Helsinki, Finland between 1967-2008 to track their structure and change over time.

Now Hiippala, who is currently working at the Centre for Applied Language Studies of the University of Jyväskylä, is turning his attention to Helsinki again, using scientific computing resources provided by CSC, the Finnish IT Centre for Science and research and education network FUNET:

Tourist photographs

“The City of Helsinki has given me a grant for analysing photographs shared by tourists on the Instagram photo-sharing service. The purpose is to find out what tourists wish to show to their close and extended networks when visiting Helsinki”.

“I’m currently working on cleaning the data, as Instagram is cluttered with memes and screenshots, and on distinguishing between locals and tourists. After I’m done, my plan is to identify the best algorithms for grouping similar images.

“My reason for working with the photos taken by tourists is that we already have many theories about the tourists’ photographic behaviour – what I want to find out is whether we can also find empirical support for these theories”.

Too much data

“The obvious problem here is that there’s simply too much data for the human analyst to handle – that’s why I’m bringing in computer vision and the infrastructure provided by CSC”.

In the future, Tuomo Hiippala plans to explore how to use computer vision to automatically identify components of page-based documents and to partially automate the annotation (description) process.

This would considerably speed up the process of building corpora, and facilitating empirical research on the complex phenomenon that is multimodality.

Above: A photo from the Finnish archives to which Tuomo Hiippala has applied some basic computer vision techniques, ie. identifying keypoints and matching them across images