Grant: Discovery Grants Program - Individual

The automation of exposing fake images and videos is gaining more and more momentum. With the exponential growth of visual assets and the more and more sophisticated tools of content editing, there is a real need of developing frameworks to support the chain of custody of media content to preserve its probative value. Unmasking image forgery remains very challenging. In the literature, the active and passive tampering footprints-based methods are the most prevalent. The active forms are mostly two-phases, first they insert a snippet (watermark/signature) in the image during acquisition or before external sharing. In a second phase, a reverse process permits to juxtapose and collate together the origin and the embedded snippet. On the contrary, the passive methods do not need any add-on information. They merely rely on the fact that a doctored image may contain measurable traces of manipulation. Bi-linear models for solving two-factor problems were widely adopted in different areas, such as separating style and content in white and black images (e.g., disentangle alphabet shape from calligraphy styles). Natural images need more complex multilinear models to cope with different factors (e.g., decoupling facial expressions from identity, head pose, and illumination). In this research, I'm seeking to design variational autroencoder (VAE) models to deal with the difficult problem of decoupling the constituent factors of the image. The different noise, artifact and mosaicking styles can be reversely recovered by particular adversarial architecture-based models, which are dedicated to style transferring between paired and unpaired images. These models are good for inducing latent codes for style separation. My motivation is that VAE models offer a neural net architecture within a high-order probabilistic graphical framework. My objective, is to use the unified framework to learn a latent representation directly from the image, aiming to disentangle the tempering profiles from doctored images. My assumption is that the traces of manipulation (e.g., local noise distribution, artifacts, foreground/backgrounds, scene lighting etc.) around the tempered region are different from those of the rest of the image. In terms of applications, the proposed framework could find uses at either individual, organizational, and societal levels. As an immunizer against credible yet fictitious media, it can be used by armed forces for combating any propaganda for sapping the troops' moral, by government to timely respond to domestic massive disinformation (e.g., real-time unmasking for an instant reply to a last-minute defamatory election campaign). It can be also used by the news media to verify the integrity of the assets before any mass public spreading. In courtroom, such applicable solutions would suggest digital corroborating evidence (e.g., verified surveillance video). It would even give a decisional certainty to the proof.