Computer Vision for Art and the ECCV conference in Munich

Check out the original paper:  “Deep Transfer Learning for Art Classification Problems”: M. Sabatelli et al. European Conference on Computer Vision (ECCV), 4th Workshop on Computer VISion for ART Analysis (VISART IV), München (GE). Code, data and camera-ready manuscript are available from Github.

The research which has so far been done in the Walloon part of the country has investigated whether state of the art algorithms that come from the fields of machine learning and Computer Vision (CV) could also be used when dealing with images representing Digital Heritage. If on the one hand, the field of CV is a well established research domain, which over the recent years has shown the effectiveness of algorithms such as Deep Convolutional Neural Networks (DCCNs), the potential of such neural architectures in the field of Digital Heritage is still an open question. Thus we have performed an empirical study that aimed to answer this research question. We have investigated a set of different strategies that can be used for training the neural networks, which, once established, have led to the conclusion that DCNNs can be a powerful tool for automatically classifying heritage objects. Besides providing an interesting support tool for art historians who aim to classify artworks, we have also tried to provide insights about why such neural networks can perform so well if properly trained, with the goal of making such algorithms understandable to the people who are less familiar with the field of machine learning.

The results of this research have led to the publication of the first CV article within the INSIGHT project called “Deep Transfer Learning for Art Classification Problems”. The paper [1] has been presented on the 9th of September at the European Conference on Computer Vision (ECCV) in Munich, during the VisArt workshop, a bi-annual event which gathers international researchers working at the intersection between Artificial Intelligence and cultural heritage. The paper has been reserved a 20 minutes talk during the workshop’s morning session called “Deep in Art”, and has been successfully accepted by the AI community.

This German experience has seen our PhD student Matthia leaving Liege at 5 in the morning of September 7th on the way to the south of Germany by train. Once settled down in his Airbnb, it was already time for him to start attending the first day of workshops at the conference, and to scientifically investigate whether German beers are as good as they say.

After having assessed the superiority of Belgian beers over the German ones, on September 9th the paper has been presented at the workshop in front of over 150 people. Besides being accepted with a lot of enthusiasm by the people attending the workshop, the paper has also caught the attention of the Google-AI headquarters in Munich. In fact they have invited Matthia to attend a special barbecue, hosted in their offices, that has given our PhD student the chance to advertise the INSIGHT project inside the biggest tech company in the world.

In the past months, thee amazing new team members started on the project. Below, they introduce themselves and their work.

Matthia Sabatelli (University of Liège)

I am working on the INSIGHT project from the Walloon part of the country where I am currently a PhD candidate in Machine Learning at the University of Liege, under the supervision of Dr. Pierre Geurts. Before that, I received my Master of Science in Artificial Intelligence (AI) from the University of Groningen in the Netherlands. My main research interests lie in the study of Deep Artificial Neural Networks where I am trying to gain insights about what makes these complex algorithms be such a successful and powerful tool in the fields of AI and Computer Vision. This understanding is then used within the INSIGHT project to build a new generation of machine learning based tools that can enrich heritage collections. Next to that, I am fascinated by the capabilities that Deep Neural Networks have in developing the most varied creative skills, ranging from the generation of artificial art, to the evolution of creative strategies that allow them to master video and board-games. More particularly, I investigate these creative processes from a Deep Reinforcement Learning perspective where I try to understand what makes Deep Neural Networks develop such original policies.

Odile Keromnes (Royal Museums of Fine Arts of Belgium)

Odile Keromnes is a young half-French half-German photographer, who lives in Brussels. Her two main passions are photography and museums. For that reason she studied art history at the university before entering the Ecole Nationale Supérieure Louis-Lumière in Paris, where she did a Master’s Degree in Photography. In this technical school, she specialized in cultural heritage digitization, doing internships in the C2RMF laboratory at the Louvre and in the photographic service of the Centre Pompidou. Now she is able to combine her two passions, working for the Royal Museums of Fine Arts of Belgium as a technical specialist for the Photographic Service. She will contribute to INSIGHT by tackling the more technical aspects of cultural heritage digitization. Her background in art history will allow her to do quality control on the first results of the AI algorithms. She will participate in the encoding of missing data for the pilot. Her work will also include the digitization of more works of art, which will enrich the project with new pictures, giving the AI more material to analyze.

Nikolay Banari (University of Antwerp)

I studied applied mathematics and physics at Moscow Institute of Physics and Technology and artificial intelligence at KU Leuven. When I was a student, I taught mathematics and physics on a part-time basis for children seeking admission to universities. Also, I worked as a Data Analyst at Forum Investment Company. Nowadays, I am a PhD researcher at University of Antwerp. I have research experience in image processing. My general scientific interests lie with the intersection of machine learning and humanities. I find the idea of applying machine learning algorithms in humanities very promising.

• The recently started BELSPO-funded INSIGHT project (Intelligent Neural Systems as Integrated Heritage Tools) organizes a launch event on 9 November 2017. This event will take the form of an afternoon of plenaries by internationally recognized speakers on topics relating to Artificial Intelligence, Heritage data and Digital Art history. This afternoon will take place at the Musical Instruments Museum in Brussels (Hofbergstraat 2, Brussels). Afterwards you are cordially invited to a reception. Registration is free but participants are invited to register through sending an email to mike.kestemont@uantwerp.be. The presentation of the event will be in the hands of Bart Magnus (PACKED).

Schedule

Welcome (Mike Kestemont) [slides]

13:00-13:45	Seth van Hooland (Université Libre de Bruxelles): Understanding the perils of Linked Data through the history of data modeling
13:45-14:30	Benoit Seguin (École Polytechnique fédérale de Lausanne): The Replica Project: Navigating Iconographic Collections at Scale
14:30-15:15	Roxanne Wyns (KULeuven): International Image Interoperability Framework (IIIF). Sharing high-resolution images across institutional boundaries
15:15-15:45	Break
15:45-16:30	Saskia Scheltjens (Rijksmuseum Amsterdam): Open Rijksmuseum Data: challenges and opportunities
16:30-17:15	Nanne van Noord (Universiteit Tilburg): Learning visual representations of style
17:15-18:30	Reception

---

Understanding the perils of Linked Data through the history of data modeling

Based on concrete examples from the cultural heritage world, this talk illustrates how different data models were developed, each one offering its own possibilities and limits. Putting RDF in this larger context helps to develop a more critical understanding of Linked Data.

Seth van Hooland studies document and records management strategies and enjoys cleaning up dirty metadata. He is the deputy head of the Information and Communication Science department and responsible for the Master in Information Science at ULB.

---

The Replica Project : Navigating Iconographic Collections at Scale

Benoit Seguin

In recent years, museums and institutions have pushed fBenoit Seguin
or a global digitization effort of their collections. Millions of artefacts (paintings, engravings, sketches, old photographs, …) are now in digital photographic format. Furthermore, through the IIIF standards, a significant portion of these images are now available online in an easily accessible manner. On the other side of the spectrum, the recent advances in Machine Learning are allowing computers to tackle more and more complex visual tasks.

The combination of more data and better technology opens new opportunities for Art Historians to navigate these collections. While most of the time search queries are solely based on textual information (metadata, tags, …), we focus on visual similarities i.e working directly with the images.

I will quickly report on our digitization effort with the processing of the fototeca of the Foundazione Giorgio Cini, where more than 300’000 images are already digitized. Then I will explain how we worked with Art Historians to learn a specific image similarity function by leveraging the concept of Visual Links between images. Finally, I will present our work on the user interfaces leveraging such a metric, allowing new ways to explore large collections of images, and how can users further refine the tool.

Benoit Seguin is a PhD Candidate at the Digital Humanities Laboratory of EPFL. Prior to starting his PhD, Benoit got his Diplôme d’Ingénieur at Ecole Polytechnique ParisTech and a MSc in Computer Science from EPFL. Before applying Image Processing and Computer Vision to Digital Humanities, he was using similar techniques in Circuit Manufacturing, Robotics, and Medical Imaging. The main topic of his PhD is to use Machine Learning to explore the next generation of tools for Digital Art History.

---

Open Rijksmuseum Data : challenges and opportunities

---

Learning visual representations of style

An artist’s style is reflected in their artworks, independent from what is depicted. Two artworks created by the same artist that depict two vastly different scenes (e.g., a beach scene and a forest scene) both reflect their style. Stylistic characteristics of an artwork can be used by experts, and sometimes even laymen, to identify the artist that created the artwork. The ability to recognize styles and relate these to artists is associated with connoisseurship. Connoisseurship is essential in the tasks of authentication and restoration of artworks, because both tasks require detailed knowledge of stylistic characteristics of the artist. In this talk I will cover a number of studies we performed aimed at realising connoisseurship in a computer, for both recognition and production tasks.

Nanne van Noord is a Postdoc working on the SEMIA (The Sensory Moving Image Archive) project at the Informatics Institute, University of Amsterdam. Previously he was a PhD candidate working on the REVIGO (REassessing VIncent van GOgh) project at Tilburg University. His research interests include deep learning, image processing, and digital cultural heritage.

---

International Image Interoperability Framework (IIIF). Sharing high-resolution images across institutional boundaries

Roxanne Wyns [slides]

IIIF or the International Image Interoperability Framework is a community-developed framework for sharing high-resolution images in an efficient and standardized way across institutional boundaries. Using an IIIF manifest URL, a researcher can simply pull the images and related contextual information such as the structure of a complex object or document, metadata and rights information into any IIIF compliant viewer such as the Mirador viewer. Simply put, a researcher can access a digital resource from the British Library and from the KU Leuven Libraries in a single viewer for research, while allowing the institutions to exert control over the quality and context of the resources offered. KU Leuven implemented IIIF in 2015 in the framework of the idemdatabase.org project and has since been using it in a number of Digital Humanities projects with a focus on high-resolution image databases for research. By now the IIIF community has grown considerably with institutions such as the ‘The J. Paul Getty Trust’ and the ’Bibliotheca Vaticana’ implementing it as a standard and providing swift and standardized access to thousands of resources. Its potential has however not reached its limits with ongoing work on aspects such IIIF resource discovery and harvesting of manifest URLs, annotation functions and an extension to include audio-visual material. The presentation will introduce IIIF and its concepts, highlight projects and viewers, and give an in-depth view of its current and future application options.

Roxanne Wyns studied History of Art and Archaeology at the Free University of Brussels (VUB). Since 2009 she worked on several European projects, specialising in standards, multilingual thesaurus management, and data interoperability and aggregation processes. At LIBIS – KU Leuven she supports KU Leuven and its partners in realizing their digital strategy. She is involved in several research infrastructure projects and supports the ‘Services for researchers’ in the framework of Research Data Management (RDM). Roxanne is actively involved in the IIIF community and is a member of the Dariah-EU Scientific Advisory Board.