Grant: Discovery Grants Program - Individual

The modern world relies heavily on the correct operation of communication networks in general and the Internet in particular. Rogue actors regularly attempt to disrupt such networks, comprising confidentiality, availability, and/or the integrity of essential information. These rogue actors either are motivated by monetary gains (blackmailing the owners or operators of compromised systems) or political considerations. Recognizing this fact, organizations and governments are interested in hardening their systems, both by learning about possible attack vectors, and deploying suitable defenses. This is typically carried out as contests between a "red team" attacking the network and a "blue team" defending it, assembling a "game" between the red and the blue teams. A key challenge is the scalability of the expert red and blue teams. Even though cyber defense and attack emulation tools provide the red and blue agents tools to automate the IT workflow by enabling the staging, scripting commands and filling in the payloads, the sequential decision making at each action step still relies on the human cyber expert. Machine learning (ML) in general, and reinforcement learning (RL) in particular, hold great promise for both red and blue agents, potentially going well above simple automated tools like that for penetration test. Depending on the sophistication of the RL training environment, trained agents may exhibit human-level or even superior intelligence in the choice of their actions. It is therefore essential for operators of networks and government agencies tasked with defending a nation's cyber infrastructure to explore these capabilities and to build appropriate defenses. At the RL core is a training environment that both mimics the real world and allows agents to be trained. Recently, for autonomous cyber operations, several such training environments have been proposed and described in the literature. All environments provide interfaces to standard RL frameworks, such as OpenAI Gym, to bring advances in RL and deep RL learning, into the cybersecurity domain. Simulation-based environments typically are abstract and high-level, making it difficult to transfer a trained agent into real networks. Emulation-based training environments usually suffer from very slow learning rates, limiting their use to small-scale networks. We are developing a general methodology and, in collaboration with research partners such as DRDC, are implementing a training and evaluation environment for cyber agents that 1.efficiently trains agents for non-trivial scenarios, 2.results in agents that can transfer into real networks directly, and 3.generalizes agents to perform well across a range of scenarios.  In this proposal, we focus on the third long-term goal. We propose to develop novel solutions based on Transfer Learning and Lifelong Learning, applied to RL, which will allow the agents to evolve over time, rather than repeatedly training them from scratch.