DEArt: Building and evaluating a dataset for object detection and pose classification for European art

Abstract

Annotation of cultural heritage artefacts allows finding and exploration of items relevant to user needs, supports functionality such as question answering or scene understanding, and in general facilitates the exposure of the society to our history and heritage. But most artefacts lack a description of their visual content due to the assumption that one sees the object; this often means that the annotations effort focuses on the historical and artistic context, information about the painter, or details about the execution and medium. Without a significant body of visual content annotation, machines cannot integrate all this data to allow further analysis, query and inference, and cultural institutions cannot offer advanced functionality to their users and visitors. Given how time-consuming manual annotation is, and to enable the development of new technology and applications for cultural heritage, we have provided through DEArt the most extensive art dataset for object detection and pose classification to date. The current paper extends this work in several ways: (1) we introduce an approach for generating refined object and relationship labels without the need for manual annotations, (2) we compare the performance of our models with the most relevant state-of-the-art in both computer vision and cultural heritage, (3) we evaluate the annotations generated by our object detection model from a user viewpoint, for both correctness and relevance, and (4) we briefly discuss the fairness of our dataset.