Category: Uncategorised

INSIGHT aims to use recent advances in Artificial Intelligence (language technology and computer vision) to support the enrichment of art collections with descriptive metadata. Multilingualism is a crucial research aspect in this respect. For the Royal Museums of Fine Arts of Belgium it is very important to be able to offer the entirely bilingual collection database (DU – FR)  in a trilingual way (DU – FR – EN). The INSIGHT project responds to this by focusing on automatic title translation of works of art.

It is not easy to tackle title translation within the art sector, even for experienced translators. Deploying machine translation for this purpose is effectively a major challenge. In some cases, visual data can help (via multimodal learning, combining computer vision and natural language processing). In this respect, we mainly think of old masterpieces with descriptive titles for more realistic scene depiction. For modern and contemporary art, the use of visual context becomes more difficult, as the link between title and work of art is often less obvious.

Methodology

The Departments of Linguistics and Literature at the University of Antwerp, which deals with these issues in the project, obviously does not aim to compete with commercial translation masters who have unlimited computer power. The team wants to tackle smaller, domain-specific datasets and experiment with other translation techniques such as ‘character level translation’ and ‘multimodal learning’. A number of papers were recently published in this area, setting the stage for the application of these methods “in the wild”, i.e. for actual digital collections in the GLAM sector. Character-level machine translation seems very promising in this respect, although this is computationally challenging. Interestingly, we can directly inject domain-specific knowledge that is available about artworks, such as IconClass-codes, to improve the results .

---

Within our specific subproject, Dutch titles form the basis for the English automated translations. The algorithms need to be trained with sufficient data for this purpose. They were pre-trained on the large EuroParl corpus and then trained with RKD datasets and some 1000 titles from the RMFAB collections which are already have available in English. Initial tests show that, as was to be expected, a much as possible sector-specific training data is needed to achieve an optimal result.

Challenge in Consistency

In order to be able to provide additional material to help raise the learning curves of the algorithms, the Museum needs to review its internal linguistic conventions (e.g. to use gender neutral terms: De Baadsters -> Bathers; to avoid initial articles, unless it is commonly used and confusion could result if it were omitted: e.g. Het meisje -> Girl; to use Self-Portrait instead of Autoportrait…) and to apply those to the existing datasets and new data input to deliver. One cannot expect the algorithms to deliver good quality if the training data shows inconsistencies. The consistent application of internal conventions is moreover important both in the initial allocation of titles (in DU and FR) and in the translation of titles.

In addition to cleaning up the existing datasets, the regulatory framework will form the basis for the volunteers who are willing to translate large quantities of titles into English in order to provide the algorithms with new training material. It is, of course, essential and inevitable that future automated title suggestions will have to pass the all-seeing eye of the curators. They will make the final adjustments to arrive at official titles validated by the Institution. Through INSIGHT we hope to, at least, facilitate the translation process by offering a first basic translation.

By: Lies Van de Cappelle (Royal Museums of Fine Arts of Belgium, Brussels)

Recently, the INSIGHT project was invited to present its work in the DH lunch lectures series organized by our kind colleagues in Krakow. Because of the global lockdown, we presented our work virtually via a prerecorded session (archived link), followed by a live Q&A session over videoconferencing. Those who missed the talk are welcome to watch the recording which was published on YouTube afterwards. As will become clear in the talk (also embedded below), we discussed our work on Minerva, in which we capitalize on object detection methods from computer to detect musical instruments in unrestricted artworks from the visual artworks. We have an awesome paper coming up on this subject in a special issue of the journal Digital Humanities Quarterly and we’ll make sure to blog more extensively about that once it’s out — any day now. Stay tuned!

In this blog post, project member Marie Coccriamont writes about her experiences and adventures collecting and annotating images of mammals in the visual arts.

A new challenge

After the annotation process of musical instruments (a blog post will follow soon!) we searched for a new annotation subject to further continue the INSIGHT project.

Animals in art have been a popular study topic throughout the past centuries. They are represented within a wide range of art objects and art genres and have either a central or peripheral role in the composition. Animals can function as complemental elements in landscapes, genres or religious scenes, they feature prominently as heraldic charges, they have been important subjects in sculpture, etc. These representations can have different levels of meaning and can assume a symbolic function in expressing the urges of society from a particular period in time and from a particular culture.

The previously mentioned properties show that this topic can serve as an interesting and useful research subject for the INSIGHT project. However, a feasibility study identified a number of problems. What to do with the representation of beasts and mythical animals? What to do with the variety of birds, the variety of dog breeds, the variety of insects, … A demarcation on the subject was needed. Because of the specific aim of the project, specific and clear labels were necessary. With the guidance of Imagenet and Wordnet, which provide a reliable structure, we came up with a new subject: ‘mammals’. With the help of standard ontologies such as Garnier and IconClass, a research of the most common mammals in the collections of KMSKB and KMKG, and the analysis of the symbolic value of mammals throughout art history, 25 labels were chosen. Regarding these 25 labels, images of art objects were gathered from the collections of KMSKB, KMKG, Wikimedia and Rijksmuseum.

Where to get data?

A first annotation cycle was done on the basis of the collections of the KMSKB and KMKG. With the help of the existing thesaurus of both collections, images of art objects of the general category “mammals”, were collected, without taking in account the 25 chosen labels. Therefore an overview of the representation of mammals in the collections was possible. An additional collection cycle was performed using the Wikimedia and Rijksmuseum collections. I gathered at least 100 pictures of art objects per label from both collections. Therefore I could ensure an equal number of images per label.

I started off with the annotation of the collections of the KMSKB and KMKG. I came across a massive number of horses, domestic dogs, cows and sheep. The number was that big that I had to delay the annotations of those mammals to guard the equality in relation to the other mammals (see figures). Other problems that occurred, were difficulties with distinguishing mammals. In one of the examples below, we see a painting in which several horses are pulling a carriage. Because of the angle of the painting, it is difficult to annotate each separate horse using a rectangle. In yet another example, we see a painting with the representation of a group of cows and a herd of sheep. Because of the amount of cows and sheep and the bad quality of the image, it is difficult to distinguish each separate mammal.

More difficulties…

Another interesting difficulty was the representation of “fantasy” mammals. Mammals that are abstracted using the imaginative mind. Examples of these “fantasy” mammals can be found in the example below. Because of the huge unambiguity associated with these animals, I chose not to annotate these images.

Another difficulty that I came across, was a low visibility or an ambiguity of the representation of mammals. This can be related to the bad quality of the medium, the abstraction of the mammal, the angle of the art object, … You name it!

The large amount of low-quality quality images made the annotation process more difficult:

Also, and as could be anticipated,  I also experienced difficulties with labelling certain mammals. It was only during the annotation process that I realized how certain labels were too closely related to each other. Bull and cow for example, are difficult to distinguish. Rat and mice also have much similarities:

Finally, I came across several images that used mammals “as a metaphor”. Such personifications of mammals can be seen in the following images. In these images I doubted how to annotate the mammals. I choose to annotate only the head of the mammals, because the bodies are often those of humans.

We thank Marie for her huge contribution to the project and we are looking to what we will get out this wonderful data in a next phase of the project!