Nathan Yong

A-EYE IN THE SKY: Examining Machine Learning-Based Keystroke Reconstruction from Passive Video Capture

Started May 29, 2025

Abstract or project description

The rapid growth of artificial intelligence (AI) has driven breakthroughs across disciplines, but has also introduced new cybersecurity threats. Among these are machine-learning (ML) powered side-channel attacks. In this paper, we examine ML enabled optical side-channel attacks and assess the feasibility of vision-based keystroke inference as an automated attack vector. Framing the problem as a supervised classification task, we systematically explore a minimal Neural Network (NN) pipeline. We compare ten critical hyperparameters through 26 models. Novel variations are explored and we produce an optimal variation within this framework. Our findings demonstrate that effective keystroke inference can be achieved with modest computational resources and without sophisticated preprocessing or postprocessing. This suggests such attacks can be utilized by moderately skilled adversaries, highlighting the need for countermeasures to such attack vectors. This research offers reproducible insights on architecture hyperparameters, along with hypotheses explaining observed phenomena, thereby advancing the study of ML enabled side channel attacks.