Team: Max Del Rio, Ben Patton, & Dylan Wheeler

Sponsor: Ken Graf

This document outlines definitions, diagrams, and technical specifications pertaining to the REDSWARMS capstone project for the UNH Cyber Security Club.

Terms and Definitions

• REDSWARMS Resilient Enterprise Deployment System Which Automates tRaining Modules Securely
• UNH CSC - University of New Hampshire Cyber Security Club
• Ansible - An automation tool which can be used to deploy applications, configurations, and manage multiple computer systems from a single node
• Ansible Tower AWX - The management console used within the CSC network to control deployments, referred to henceforth as just ‘AWX’
• Red Team - A common team present during cybersecurity competitions, the red team is a group of ‘attackers’ trying to penetrate and tear down the competition network and hosts
• Blue Team - The other common team present during competitions, the blue team is responsible for defending their systems from the red team and keeping scored services active, while also providing reports to the white team
• White Team - A team which represents upper management and company executives during competitions.
• Black Team - The team responsible for setting up and maintaining infrastructure and software for a cybersecurity competition
• DMZ Demilitarized Zone - Commonly used in enterprise networks to host outward-facing services and computers, such as web servers or customer support systems.
• Scoring Engine - A piece of software used in competitions to test if certain network services are available within a blue-team-defended network. Will typically send pings, HTTP GET messages, or TCP messages to certain well-known ports to test connectivity.

Overview

As the world becomes more digital, cybersecurity becomes ever more vital. To anticipate this demand in talent, the University of New Hampshire has a cybersecurity club and team that fosters growth in students by training them to face real-world scenarios through competitions. However, building these mock networks for training is currently manual, time-consuming, and error-prone. REDSWARMS is a system designed to facilitate the automatic creation and configuration of small-scale enterprise networks to more quickly perform those security training scenarios and competitions. It features virtual infrastructure using VMWare’s platform for managing a network of virtual machines as well as an automated deployment pipeline with Ansible and Ansible Tower AWX. With REDSWARMS, the cybersecurity team can stand up brand new networks for training in a matter of minutes, allowing students to spend less time configuring and more time learning.

Requirements

The main goal of the REDSWARMS project is to make setup and configuration of training networks fast and efficient. For future use, the system also needs to be architected in a way that would facilitate simple additions and modifications to team network deployments. With respect to security, the system needs a secure way of creating and managing credentials for both the static systems used for deployment and for the virtual systems being deployed. It should be noted that all virtual systems being deployed by REDSWARMS are left intentionally vulnerable in some way for training purposes and that any security for those systems is maintained via their segregation from the rest of the UNH CS department network.  Finally, the system needs to be well-documented and simple enough to allow new club and team members to be able to simply use it. This documentation will likely take the form of a wiki available to the team that explains key components of the CSC lab network, the tools and services required to deploy networks, and how to write scripts that are used in the deployment process. The following table outlines the MOV for the REDSWARMS project:

Cyber Security Lab Network Architecture

The Cyber Security Lab network mirrors an enterprise network. Within the Lab Network, we have control over a firewall/router, physical servers, and a virtual computing cluster (VMware vCenter). The Lab features its own Active Directory server serving all users authentication within the local lab network.

One of the most important components of the CSC lab infrastructure is a segregated and firewalled virtual network intended for hosting competitions. In a competition setting, this network is designed to simulate a WAN network to individual teams and has thus been dubbed the Internot by the UNH CSC. The main router for the Internot is a virtual Palo Alto appliance, which has interfaces connecting it to several different subnets within the CSC network:

• BlackNet - Used for management of key hosts and infrastructure by the CSC
• Internot - Simulated WAN network for competitions, will also host any computers and support systems used by the red team
• GreenNet (AKA Team Routing Net) - Subnet with a limited address range used for connecting individual team routers to the Internot router

The diagram below gives a visual representation of the CSC network as it relates to key management hosts/routers and to competition team networks:

The main address range given to each team network corresponds to their team number (used as the third octet of their address range). Each team network is divided into two smaller LAN and DMZ subnets with a 25-bit network prefix.

Competition Team Network Architecture

During competitions, the UNH Cyber Security Team deals with network devices and architectures that may be found in use at a real business of varying sizes. Therefore, the REDSWARMS deployment system needs to deploy networks with similar systems.

The network infrastructure for a single team consists of three distinct subnets/zones:

• WAN
• DMZ
• LAN

The LAN network contains all client devices, as well as an Active Directory/DNS server and a management device. The client devices on the LAN network are a mix of Linux and Windows hosts. The Linux hosts will most likely be running desktop distributions of CentOS and Ubuntu, while the Windows hosts will be running Windows 7 and 10.

The DMZ network contains all devices that would theoretically be serving content and services available to external parties over the Internet. This may include:

• Web Server (running Nginx on Ubuntu Server)
• Database Server (running PostgreSQL or MySQL on Ubuntu Server)
• Email/FTP Server (running on Windows Server 2019 or 2016)

The WAN network is not used directly by the competition team, but all network services that the team is scored on will need to be accessible from the WAN. A separate node running a competition scoring engine will be placed on the WAN, and will periodically test each node on both the DMZ and LAN for their respective service. Typically graded services may include SSH or ICMP on clients, and HTTP or SMTP connectivity on servers, as well as other management and content-serving protocols.

Codebase

The main code base for the REDSWARMS project is kept under version control using Git with a central repository hosted on a bitbucket instance running in the cloud. The codebase consists almost exclusively of Ansible scripts, called ‘playbooks’, which are read and executed by the controlling Ansible Tower AWX instance to automatically create and configure virtual machines in the CSC VM cluster. For more information about the contents of the codebase, see the relevant section in the appendix of this document.

Deployment Pipeline

Automated deployments are made possible through the use of Ansible, an open-source deployment and management tool. Ansible is capable of creating and configuring each host on the virtual network as specified in YAML markdown files called ‘Playbooks’. The hosts involved in each deployment are specified in a separate ‘Inventory’ YAML file and include variable definitions for information such as the VM template files, users and hostnames, and network interface and IP address configuration. Those variables are then supplied to playbooks, which run tasks for configuring the hosts. Those tasks can be anything from basic system configuration to installing and running applications.

Inventory Hierarchy and Definition

Here is the hierarchy of how everything in Ansible is read when a deployment is started:

• Deployment Template
• ‘Master’ playbook from which to execute tasks
• Project, which is a git repository contains playbooks used in this template
• Inventories
• Groups of hosts
• Individual hosts

At the inventory level, variables can be defined in a ‘hosts’ file, and are then inserted into the deployment playbook at runtime. Variables for our system that are defined at the inventory level pertain mostly to the VM cluster that deployed machines will live on. Things such as:

• VMWare host, username, and password
• Location on VMWare cluster to which the inventory hosts will deploy
• Some higher-level network interfaces

Playbook variables can also be defined at the host or group level. For our system, this is where competition team-specific information can be defined, both for individual virtual machines and for different subnets within the team network. Some typical variables specified here include:

• VM name
• VM template file
• VM network adapter (LAN, DMZ, etc.)
• VM IP address and subnet mask

Currently, there are inventories completed for Windows machine deployments and Linux machines. The completed inventories are relatively system agnostic, but both are capable of deploying and configuring multiple hosts.

All of the variables used in project inventory files are documented further online in Ansible’s documentation

Inventory groups are of particular use to the REDSWARMS deployment process and act as tags for storing variables that can be used by several hosts. For example, a group exists for LAN devices and a group exists for DMZ devices. Individual hosts in an inventory that are “tagged” as being in the LAN group get access to LAN-specific network configuration variables, whereas hosts “tagged” for DMZ will get the same network variables but specific to hosts on the DMZ network. Currently, hosts can be “tagged” as being in the following groups:

• LAN
• DMZ
• Linux host
• Windows host

This grouping allows for a flexible and logical system for supplying variables to playbooks during team deployments.

Playbook Execution

The deployment process for a given virtual host in the REDSWARMS network is to first clone the virtual machine from a template and tailor certain hardware and network settings, then to install and configure relevant applications on the VM itself. The VM cloning and configuration of the hardware and network specs are handled by the clone_vms.yaml, after which a service-specific playbook is invoked by the configuring node to access the host, download relevant packages, create or write to app config files, and start services. This playbook execution process is frequently used in the REDSWARMS system.

A user can deploy a team network or invoke a specific step in the deployment process via the Ansible AWX interface by launching a template. Each template has an associated playbook and inventory, although for a given scenario the inventory will be common among all templates. The REDSWARMS system selects applicable hosts in the specified inventory through the use of the ‘tagging’ system described in the Inventory Hierarchy and Definition section of this document.

Ansible AWX allows for all templates to be connected and executed sequentially through the use of a template workflow. The graphical workflow editor allows for branching paths within the deployment process, and certain steps can be executed contingent on the success of previous steps. In the diagram below (a screenshot of the workflow editor), each step is shown as a given Ansible template. Organizing deployment and configuration in this way allows for more flexibility, as templates can be added or moved easily without affecting other steps in the team deployment.

Status

The project team’s JIRA board can be viewed and referred to for progress on the project. Issue identifiers have also been provided in this document where relevant.

Completed Work

Network Configuration

The network in the CSC lab has been architected to support the creation and deletion of new team network subnets, through the creation of the Internot and GreenNet.

There has also been considerable effort put into the setup of VM infrastructure that will support the deployment of competition networks, such as the creation of an Ansible Tower AWX instance and the Internot virtual router appliance.

Software and Scripts

Code exists in the project team’s MainCode repo for several different host and service deployments:

• Windows 7/10 Desktop - REDSWARMS-58
• Windows Server with AD service - REDSWARMS-57
• Web server running nginx - REDSWARMS-14
• Server hosting a Redmine issue tracker - REDSWARMS-16
• pfSense router
• Ubuntu desktop

Ansible playbooks also exist for controlling various vCenter functions:

• Creating team folders - REDSWARMS-26
• Cleanup for a competition network - REDSWARMS-28
• Creation of virtual port groups (virtual LAN and DMZ interfaces) - REDSWARMS-27

Documentation

At this stage, we have completed our competition packet as well as a work-in-progress network diagram. The packet will be used to inform teams when they arrive for UNH-sponsored competitions. The network diagram helps participants understand which machines correspond to which IP addresses. Team packet issue is identified on JIRA as REDSWARMS-32.

Research

Testing has also been completed to determine the hardware performance on the VRTX hosting our deployed VMs. According to IOMeter results, a RAID array composed of Solid State Drives instead of Hard Drive Disks could decrease deployment time by a factor of eight. Progress on this investigation can be tracked on JIRA as REDSWARMS-52.

Unfinished and Future Work

Research

• REDSWARMS-76 - Investigate Vulnerable Systems
• Once single team networks can be deployed reliably, the team will need to research common software and OS vulnerabilities that can be automatically configured on virtual machines as part of a deployment.

Software and Scripts

Additional software and Ansible scripts that need to be developed:

• SQL database server - REDSWARMS-55
• Scoring Engine - REDSWARMS-25

More scripts for configuration and testing will likely need to be implemented once the team has determined what vulnerable systems should be deployed, and after further usability testing with the CSC and competition team.

Documentation

Most of the development and setup so far has gone relatively undocumented. The following stories are currently in the backlog for documenting the use of REDSWARMS for future CSC members:

• REDSWARMS-89 - Document how to use AWX
• REDSWARMS-90 - Document CSC network
• REDSWARMS-91 - Document how to create new competition scenarios

Current MOV

A full team network deployment is now possible with the REDSWARMS system, and it can also be used to deploy individual pre-configured virtual machines of various types. VM templates exist on the CSC virtual infrastructure for Windows desktop hosts, two different Windows server versions (2016 and 2019), CentOS 8 desktop, Ubuntu 18.04 desktop, Ubuntu 18.04 server, and a pfsense router. All of these templates can be deployed and given IP addresses using AWX.

Appendix

Section A: Codebase Contents and Hierarchy

The root directory of the MainCode repo contains subfolders for each type of host that can be deployed using REDSWARMS, first by host OS, then by type of service running on that OS. vcenter directory contains certain scripts for performing vcenter administrative functions, such as creating and destroying VMs. Note that this is not a comprehensive list of project files but should give the reader an idea of how the code is organized:

maincode/

• linux/
• nginx/
• redmine/
• vcenter/
• clone_vm.yaml
• destroy_vm.yaml
• create_folder.yaml
• create_nic.yaml
• windows/
• active_directory
• windows_desktop
• routers/
• pfsense_deploy_provision
• main.yaml
• deploy_pfsense.yaml
• wait_for_vmtools_pfsense.yaml
• assign_pfsense_ip.yaml

Playbook YAML files are organized and named based on the service that they configure and test.

Some directories include an additional ‘templates’ directory, which holds files that can be copied to controlled hosts via a template module. This functionality is currently used to copy server configuration files and HTML documents to web server hosts, but can and will be extended in the future to include parameterized config files for several services like windows AD or mail servers.

Section B: Playbook Definition

Perhaps the most commonly used playbook used by the REDSWARMS system is the clone_vms.yaml file:

- name: "Clone VM from template"
hosts: VMs
gather_facts: no
connection: local
tasks:
- name: "Deploy VM from template"
vmware_guest:
hostname: "{{ VMWARE_HOST }}"
username: "{{ VMWARE_USER }}"
password: "{{ VMWARE_PASSWORD }}"
validate_certs: no
datacenter: "{{ datacenter }}"
datastore: "{{ datastore }}"
folder:  "{{ vm_folder }}"
state: poweredon
template: "{{ template }}"
name: "{{ vm_name }}"
networks:
- name: "{{ vm_network }}"
type: static
ip: "{{ vm_ip }}"
netmask: "{{ vm_netmask }}"
gateway: "{{ vm_gateway }}"
dns_servers:
- "{{ vm_dns }}"
start_connected: yes
wait_for_ip_address: yes
delegate_to: localhost

The ‘hosts’ list element specifies a host group definition (described in more detail by the inventory), the ‘template’ element determines the existing VM template that is used during the cloning process. Several other inventory-supplied variables are also supplied to set various options on the VM after cloning, such as IP address, hostname, and network adapters.

An additional play is used to configure services on the newly deployed virtual machine (or group of virtual machines). The main difference between the VM deployment and application configuration plays is that the VM deployment’s tasks must be delegated to the runner, in this case, the machine running the ansible plays is also the one supporting the VMWare vCenter app.

The following is the contents of the application play:

- hosts: VMs
gather_facts: no
- name: Install dependencies for nginx
apt:
name: nginx
state: present
- name: Configure a homepage
template:
src: index.html
dest: /var/www/html/index.html
- name: Start nginx service
service:
name: nginx
state: restarted

The task list here installs dependencies (nginx), then copies an index.html file stored in the nginx templates directory to the webserver root directory on the server machine, then restarts the nginx process.