Azure Serial Console Attack and Defense - Part 2
This blog was originally posted on Microsoft's MSRC blog on December 19, 2023, and this version is an archived/mirrored version. It was created in collaboration with Malla Reddy Donapati.
This is the second installment of the Azure Serial Console blog, which provides insights to improve defenders’ preparedness when investigating Azure Serial Console activity on Azure Linux virtual machines. While the first blog post discussed various tracing activities, such as using Azure activity and Sysmon logs on Windows virtual machines to trace serial console activity, this blog outlines how to enable logging for Azure Linux virtual machines using Sysmon for Linux to capture and how to send these events to a log analytics workspace. It also covers the tracing activity of serial console in Microsoft Defender for Endpoint logs. The goal extends beyond identifying serial console activity on Linux virtual machines, to using these artifacts to create additional hunting queries for detecting potential abuse by attackers.
Contents
Enabling logging for Linux virtual machine
Serial Console artifacts on Linux virtual machine
Hunting for suspicious operations
Best practices
Conclusion
Enable logging for Linux Virtual machine
Please note that depending on your current setup, this step might not be necessary. If your cloud computing workloads are already monitored by Microsoft Sentinel or another security monitoring solution such as SIEM (security information and event management), feel free to skip this step. This section of the blog covers how to log Azure Linux virtual machine events using Sysmon for Linux and Microsoft Defender for Endpoint (for Linux) agents to capture the Azure serial console activity artifacts.
The creation of the log analytics workspace and enabling Azure activity monitoring is covered in Part 1 of the blog. To enable logging for Linux virtual machine, install the Log Analytics Linux Agent as shown below from the Agents tab under Log Analytics workspace.
Syslog
Syslog is an event logging protocol commonly used in Linux. The Syslog daemon in Linux devices collects local events as specified and can send them to a Log Analytics workspace. Applications send messages that can be stored locally or sent to a Syslog collector. When the Azure Monitor agent for Linux is installed, it configures the local Syslog daemon to forward messages to the agent when Syslog collection is enabled in data collection rules (DCRs). The Azure Monitor Agent then transmits these messages to an Azure Monitor or Log Analytics workspace, creating corresponding Syslog records in a Syslog table. Collect Syslog events with Azure Monitor Agent using the guide here.
Sysmon for Linux
Prerequisites to install Sysmon for Linux
Ensure the Linux virtual machine is on boarded to Log analytics workspace.
Collect Syslog events with Azure Monitor Agent using this guide.
Install Sysmon using this guide on Linux virtual machine. The configuration for Sysmon for linux can be found here and install the Sysmon service with the below configuration.
Serial Console artifacts on Linux virtual machine
The evidence of accessing a virtual machine using serial console can be found in native Syslog and Sysmon for Linux logs on virtual machine. The Serial console connection happens over /dev/ttyS0
on Linux virtual machines.
Tracing activity using Syslog
Identify launching of serial console to access a Linux virtual machine, it will initiate connection on ttyS0
and show an authentication prompt with username and password to login into the virtual machine. The following query can be used to identify the same.
To identify the start of a virtual ttyS0 terminal upon the /dev/ttyS0
is enabled on Linux virtual machine, the following query can be used.
Discover successful virtual machine access using Serial console in Syslog native logs.
Find Failed login attempts to connect to virtual machine using Serial console. The number of failed attempts will also be captured in syslog logs for that corresponding user account for point of time.
The use of ‘sudo’ after a successful authentication using serial console will be captured in syslog authpriv
logs with the command being executed with ‘sudo’ and the commands with ‘sudo’ will also be captured.
Tracing activity using Sysmon for Linux
Once Sysmon for Linux is successfully installed, the log analytics workspace begins to capture Sysmon events within syslog messages that contain “Linux-Sysmon.” The query below can be used to identify all process execution events on a Linux virtual machine accessed via the serial console. Sysmon for Linux logs all ProcessCreate events on the Linux virtual machine under Event ID 1.
This Kusto query can parse all Sysmon for Linux logs for all events mentioned in the current configuration file. This query can be saved as a function (for example, Sysmon_for_linux ) to search for different events on a Linux virtual machine using event ID field. Every process on a Linux virtual machine using the serial console has a unique parent process & command line. The parent process IDs will change depending on the serial console sessions. The parent process will be “/usr/bin/bash” and parent process commandline will be “-bash.” The Kusto query below can parse Sysmon for Linux logs for Event ID 1 (ProcessCreate) to identify the processes executed using serial console Special Administration Console (SAC). However, there is a caveat to this query as it only projects immediate child processes executed from the SAC console, not the processes spawned from a shell script.
For example, the below query can identify all public network connections from the processes executed (within the context of the logged in user) on a virtual machine using serial console.
Tracing activity using Microsoft Defender for Endpoint (MDE)
Microsoft Defender for Endpoint captures Azure serial console activity for a Linux virtual machine. This activity includes logging into virtual machines via the SAC console, executing all processes, and establishing network connections using the logged-in serial console. Serial console connections to the Linux virtual machine are established on virtual console terminals with device /dev/vc/
. In general scenarios, direct serial console access to any device happens over /dev/ttyS0
. The Kusto queries below for MDE will provide information related to logged-in users, executed processes, and established network connections using serial console.
The below Kusto query for MDE will identify logged in users using serial console
The Kusto query below identifies all processes executed during the session connected via the serial console.
The Kusto query below will identify all network connections from the processes executed using serial console.
Hunting for suspicious operations
The following are a few examples by using which a defender can detect suspicious activity around Azure serial console.
Hunting suspicious operations using Sysmon for Linux event logs
An attacker can gain initial access through the Azure serial console, and any activity following this access can be identified using the serial console spawned process lineage. While there are multiple KQL (Kusto Query Language) hunting queries available to detect suspicious operations on Linux virtual machine, the below Kusto query can be particularly useful for hunting the most common techniques. Note that the below query does not cover all Linux attack techniques. However, it presents the process hierarchy that can be leveraged by security analysts to identify activity done through serial console on a linux machine.
Hunting suspicious operations using Microsoft Defender for Endpoint
Access to the serial console can be exploited to misuse known native system images to perform reconnaissance, download files, search for secrets, and perform lateral movement. The below query will find the execution of those most frequently used native images using serial console
Hunting suspicious operations using Azure Activity logs
In addition to the hunting techniques mentioned in Part 1, anomalous serial console connect activity can be identified using the following query.
Hunting suspicious operations using Boot diagnostics logs
In Part 1, we discussed how an adversary can abuse boot diagnostics logs for exfiltrating sensitive information. However, boot diagnostics logs remain a valuable source of activity identification. In instances where the logs such as Syslog are not backed up to a central location such as SIEM, boot diagnostics logs can provide crucial insights. These logs are not backed up to a SIEM due to the sensitive nature of data they might contain. However, they hold significant forensic value in cases where the logs are absent.
The following is an execution of LinPEAS (a popular privilege escalation tool) and reverse shell connectivity using Netcat. [Please note that Microsoft doesn’t endorse the usage of these tools and it’s the responsibility of readers to use these tools responsibly and ethically. Please check our terms of service before using these tools against Azure or other Microsoft infrastructure.]
Best Practices
The following is a non-exhaustive list of best practices that we recommend for keeping Azure Serial Console secure:
Enforce usage of MFA (multifactor authentication) for all the users with “Virtual machine contributor” access.
Regularly audit RBAC (role-based access control) permissions of users to ensure that the list of privileged users is up to date.
Implement continuous monitoring of the Serial Console activities for Azure virtual machines using cloud native SIEM (such as Microsoft Sentinel). Set up alerts for suspicious activities.
Establish clear and documented procedures for emergency access to the Serial Console. Ensure that these procedures are known to authorized personnel and are followed strictly in emergency scenarios.
Conclusion
While Azure Serial Console is a beneficial feature that allows developers and administrators to troubleshoot during challenging times, it can become a security liability if not properly monitored and secured. This blog highlighted several opportunities for detecting malicious activity pertaining to Azure serial console. With logging opportunities, detection ideas and hunting queries discussed in the article, protecting Azure Serial console just got easy!
References
Last updated