This paper presents an online anomaly detection framework for autonomous cyber-physical systems that integrates reinforcement learning with human-in-the-loop retraining to handle evolving system behaviors.
The framework utilizes a factorized deep Q-network with self-attention to select optimal detectors from a candidate pool based on microservice dependencies. It employs an ensemble of three statistical drift detectors that prioritize precision by raising alarms only when all agree. A pending transition buffer and 60/40 prioritized replay strategy allow operators to incorporate expert knowledge without catastrophic forgetting.
Evaluated on a connected-vehicle testbed for automated valet parking, the attention-augmented agent achieved an F1 score of 0.69, significantly outperforming single detectors which scored at most 0.11. After concept drift induced by a software update, operator-triggered retraining recovered performance to 0.65 on the new distribution while maintaining 0.69 on the prior one.