Show Me Your Hand! Computational Methods for Hand Gesture Analysis in Early Modern European Painting

Valentine Bernasconi

The research consists of the computational exploration of the painted hand pose in early modern paintings from the digitized photographic collection of the Bibliotheca Hertziana. The geographical scope of the corpus relies on curatorial choices from the original collection, mostly Italian paintings. In this work, we compare computational methods and their results with the current state of research in art history. In total, a set of three experiments are made to explore the shape of a lexicon of hand gestures, to outline recurrent patterns of combinations of hands in relation to depicted stories, and to develop new research tools. The goal is to challenge traditional approaches to the question of hands in art history, as well as to enrich knowledge on early modern painted hands. 
The first experiment directly confronts the question of the existence of a specific hand language in early modern paintings. It involves the creation and training of a machine learning model to automatically recognize hands based on existing knowledge in art history. We explore in detail the classification of the art historian Temenuzhka Dimova, before adressing the complexity of training a machine learning model for the recognition of chirograms. (A chirogram is a term introduced by Dimova in her work to refer to the iconographic configurations of the hand.)
The second experiment is a prototype that allows users to retrieve specific hands from the collection based on their own hand pose captured by the computer. Called Gestures for Artworks Browsing, the tool allows us to engage with the notion of embodied knowledge in the practice of art history, as well as the assumption of influence, inherent to most digital projects on the distant reading of artworks.
The third and last experiment consists of the understanding of gestural patterns as a network. We consider new contextual layers that are formed by the direct relation of a hand with other hands within the composition and in association with an iconography. To this end, an unsupervised clustering method was used to shape new categories based on geometric properties and using the whole collection of hands. These clusters reveal another dimension of painted hands and are used to portray common combinations of hands discussed in a primary section. These combinations are then correlated with the iconographies of the original paintings, thus shaping networks of hand combinations. A critical assessment of these networks is then made. We address the iconographic take on these hands, as well as the granularity of the information used to shape the interactive visualization in confrontation with the biases of the primary detections undertaken thorugh an automated process and on the basis of an estimation of the hand poses.