The hidden images found in masterpieces

From ancient scrolls to Dutch Golden Age paintings, technologies are discovering new clues about the world's greatest art works.

The history of art is filled with lost masterpieces, paintings and artefacts that have been destroyed, altered or even painted over by the artist and discovered centuries later. After the devastating eruption of Mount Vesuvius in 79 AD, whole libraries of ancient scrolls were turned to charcoal and are now too fragile to be unrolled. One of Rembrandt's greatest masterpieces, The Night Watch (1642), had s chopped off each side in order to fit through the door of Amsterdam's Town Hall, where it was installed in 1715. We have long accepted that works such as these might never be known to us, but in recent years this assumption has been upended.

More like this:

-       Does The Night Watch reveal a murder plot?

-       The secrets of Girl with a Pearl Earring

-       How fear shaped ancient mythology

Much as AI, or machine learning, has been woven into the rest of our daily lives, it's now taking a shot at resurrecting precious cultural artefacts once lost to the age of time. For most of history it was not unusual for an artist to reuse canvases, which could be hard to come by. One project that has grabbed headlines in recent years, led by two science PhD students George Cann and Anthony Bourached, working under the name Oxia Palus, has attempted to resurrect some of the works that are now known to us only through X-rays. Their AI-rendered images (the first was made public in 2019) aim to give back an idea of the lost painting's original colour and texture.

In one example, Modigliani's Portrait of a Girl (c 1917), they have brought into the light a woman that the artist once attempted to conceal. She is believed to be the English writer Beatrice Hastings, an ex-lover with whom he had a notoriously stormy relationship. He may well have tried to erase her memory once their affair ended in 1916 (although several portraits of her do survive).

The two researchers have converted their reconstructions into 3D prints that are being sold under the name NeoMasters. "I think these have interested people because they like things that are new," Bourached tells BBC Culture. "It would be amazing to see all these masterpieces, which exist but that you'll never see because nobody is going to get a chisel out and scrape off part of a multi-million-pound painting."

The pair are wary, however, that news headlines can oversimplify their work. "The story that grabs attention is 'AI reconstructs a painting', as if there's a button that we press and the whole process is done," says Bourached. This can give the impression that scientists are handing over complete authority to machines when, in fact, he says, "there's lots of human input along the way".

The human contribution includes gathering a dataset of works by the artist for the machine to learn their style and cleaning up the X-ray image to remove elements from the surface painting. The resulting X-ray images are "very much our interpretation of what's underneath," Cann tells BBC Culture. The pair describe it as a "sloppy process", just an experiment produced in their spare time with no funding. "One of the points we want to make is that, even with a relatively naive approach, this is what can be done," adds Bourached. "I'm hoping that other people will take this nascent field and do better things with it."

Bringing art and science together

One barrier they came up against was the limited information provided by traditional X-rays, which were first used on paintings in the 19th Century. Conservators also take samples from the canvas to find out more about materials, pigments and possible damage but newer scanning technologies allow them to get all this information without touching the work.

Five years ago, the National Gallery in London acquired new state-of-the-art scanning equipment capable of capturing so much data about a single painting that most cultural organisations are poorly equipped to make much use of it. Some are now initiating cross-sector collaborations with universities that can offer superior computing facilities and broader expertise. As part of a recent partnership with University College London and Imperial College London, called Art Through the ICT Lens, or ARTICT, the National Gallery has been producing much clearer images of Francisco de Goya's Doña Isabel de Porcel (c 1805), a fashionable young woman wearing a black mantilla (scarf). In 1980, a mysterious second portrait of a man in a waistcoat and jacket was discovered underneath. Getting a clearer image of the man has meant combining multiple scans from different regions of the electromagnetic spectrum, some of which reveal certain aspects of the painting better than others. At first, this process had to be done manually but thanks to this new research, it has now been handed over to a computer.

The UCL researchers, led by Dr Miguel Rodrigues, had previously worked on a similar project for the restoration of Van Eyck's Ghent Altarpiece (1432). The large-scale work is made up of many s, some of which are double-sided so that it can be opened and closed. The exterior s, showing the annunciation and a row of patrons and saints, were visible most of the year, but the altarpiece would be opened to reveal other religious figures and the Adoration of the Mystic Lamb on special feast days. The double-sided s have produced layered X-rays that are particularly tricky to interpret, a problem for conservators hoping to study different layers of the painting. The algorithm developed by Rodrigues and his team managed to separate them into two distinct images.

Achieving this with the murky X-rays of the Goya is a bigger challenge, as the researchers did not already have an idea of what the painting underneath should look like. Instead, they mixed up X-rays of other paintings to see if the algorithm could learn to separate them. In this case, the algorithm is known as a "black box", meaning it may produce good results but it is not always clear how. To try and understand this better, the team looked at outputs from different points in the learning process and experimented with how feeding the machine data in different ways might improve the output.

"One of the things that is particularly difficult to work out is what is good and what is better," Catherine Higgitt, Principal Scientific Officer at the National Gallery, tells BBC Culture. "We're separating an image into two hypothetical images, but what is it we're trying to get? Is that something useful to a conservator – interested in damage, cracks and subsurface details – or something that looks as close to the image at the surface as possible">window._taboola = window._taboola || []; _taboola.push({ mode: 'alternating-thumbnails-a', container: 'taboola-below-article', placement: 'Below Article', target_type: 'mix' });