Cisco discovered early evidence of potentially malicious activity on September 28, 2023, when a case was opened with Cisco's Technical Assistance Center (TAC) that identified unusual behavior on a customer device. Upon further investigation, Cisco observed what determined to be related activity as early as September 18. The activity included an authorized user creating a local user account under the username “cisco_tac_admin” from a suspicious IP address (5.149.249[.]74). Instances of this activity ended on October 1, and Cisco did not observe any other associated behavior at that time other than the suspicious account creation.
Right after...
On October 12, Cisco Talos Incident Response (Talos IR) and TAC detected what they later determined to be an additional cluster of related activity that began on that same day. In this cluster, an unauthorized user was observed creating a local user account under the name “cisco_support” from a second suspicious IP address (154.53.56[.]231). Unlike the September case, this October activity included several subsequent actions, including the deployment of an implant consisting of a configuration file (“cisco_service.conf”). The configuration file defines the new web server endpoint (URI path) used to interact with the implant. That endpoint receives certain parameters, described in more detail below, that allows the actor to execute arbitrary commands at the system level or IOS level. For the implant to become active, the web server must be restarted; in at least one observed case the server was not restarted so the implant never became active despite being installed.
The implant is saved under the file path “/usr/binos/conf/nginx-conf/cisco_service.conf” that contains two variable strings made up of hexadecimal characters.
Surprise surprise, Cisco uses nginx in their firmware...
The implant is not persistent—meaning a device reboot will remove it—but the newly created local user accounts remain active even after system reboots. The new user accounts have level 15 privileges, meaning they have full administrator access to the device. This privileged access to the devices and subsequent creation of new users is tracked as CVE-2023-20198.
Cisco assess that these clusters of activity were likely carried out by the same actor. Both clusters appeared close together, with the October activity appearing to build off the September activity. The first cluster was possibly the actor’s initial attempt and testing their code, while the October activity seems to show the actor expanding their operation to include establishing persistent access via deployment of the implant.
How bad is it?
Ths vulnerability is rated with a CVS score of 10, identified as Critical (in layman's term - game over scenario).
What needs to be done?
This vulnerability affects Cisco IOS XE Software if the web UI feature is enabled. The web UI feature is enabled through the ip http server or ip http secure-server commands.
The following example shows the output of the show running-config | include ip http server|secure|active command for a system that has the HTTP Server feature enabled:
Router# show running-config | include ip http server|secure|active'
'
ip http server
ip http secure-server
Note: The presence of either command or both commands in the system configuration indicates that the web UI feature is enabled.
If the ip http server command is present and the configuration also contains ip http active-session-modules none, the vulnerability is not exploitable over HTTP.
If the ip http secure-server command is present and the configuration also contains ip http secure-active-session-modules none, the vulnerability is not exploitable over HTTPS.
Immediate mitigation
Cisco strongly recommends that customers disable the HTTP Server feature on all internet-facing systems. To disable the HTTP Server feature, use the no ip http server or no ip http secure-server command in global configuration mode. If both the HTTP server and HTTPS server are in use, both commands are required to disable the HTTP Server feature.
Remember to save the config once executing these commands. Also take note that this command effectively disables the web interface.
Indicator of Compromise
To determine whether a system may have been compromised, perform the following checks:
Check the system logs for the presence of any of the following log messages where user could be cisco_tac_admin, cisco_support or any configured, local user that is unknown to the network administrator:
`%SYS-5-CONFIG_P: Configured programmatically by process SEP_webui_wsma_http from console as user on line
%SEC_LOGIN-5-WEBLOGIN_SUCCESS: Login Success [user: user] [Source: source_IP_address] at 03:42:13 UTC Wed Oct 11 2023`
Note: The %SYS-5-CONFIG_P message will be present for each instance that a user has accessed the web UI. The indicator to look for is new or unknown usernames present in the message.
Check the system logs for the following message where filename is an unknown filename that does not correlate with an expected file installation action:
%WEBUI-6-INSTALL_OPERATION_INFO: User: username, Install Operation: ADD filename
Cisco Talos has provided the following command to check for the presence of the implant where systemip is the IP address of the system to check. This command should be issued from a workstation with access to the system in question:
curl -k -X POST "https://systemip/webui/logoutconfirm.html?logon_hash=1"
If the request returns a hexadecimal string, the implant is present.
IP Addresses
5.149.249[.]74
154.53.56[.]231
Usernames:
cisco_tac_admin
cisco_support
Implant Delivery
Leveraging existing detections, Cisco Talos observed the actor exploiting CVE-2021-1435, for which Cisco provided a patch in 2021, to install the implant after gaining access to the device.
Read this next line twice!
Cisco Talos have also seen devices fully patched against CVE-2021-1435 getting the implant successfully installed through an as of yet undetermined mechanism.
Implant analysis
The implant is based on the Lua programming language and consists of 29 lines of code that facilitates the arbitrary command execution. The attacker must create an HTTP POST request to the device, which delivers the following three functions -
The first function is dictated by the “menu” parameter, which must exist and must be non-empty. This returns a string of numbers surrounded by forward-slashes, which we suspect might represent the implant’s version or installation date.
The second function is dictated by the “logon_hash” parameter, which must be set to “1”. This returns an 18-character hexadecimal string that is hardcoded into the implant.
The third function is also dictated by the “logon_hash” parameter, which checks to see if the parameter matches a 40-character hexadecimal string that is hardcoded into the implant. A second parameter used here is “common_type”, which must be non-empty, and whose value determines whether the code is executed at the system level or IOS level. If the code is executed at the system level, this parameter must be set to “subsystem”, and if it is executed at the IOS level, the parameter must be “iox”. The IOX commands are executed at privilege level 15.
In most instances we have observed of this implant being installed, both the 18-character hexadecimal string in the second function and the 40-character hexadecimal string in the third function are unique, although in some cases, these strings were the same across different devices. This suggests there is a way for the actor to compute the value used in the third function from the value returned by the second function, acting as a form of authentication required for the arbitrary command execution provided in the third function.
Scale of Impact
Shadowserver is tracking the number of hosts compromised due to this implant. Above is a snapshot in time of the scale of this issue.
LeakIX seems corroborate the extent of this breach
Final Words
If you have any Cisco IOS XE devices directly connected to the Internet with the web interface turned on, you should immediately start your Incident Response process.
Reference
[1] Cisco Security Advisory. (2023). Retrieved from https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-iosxe-webui-privesc-j22SaA4z
[2] Talos, C. (2023). Active exploitation of Cisco IOS XE Software Web Management User Interface vulnerability. Retrieved from https://blog.talosintelligence.com/active-exploitation-of-cisco-ios-xe-software/
[3] Dan Goodin - Oct 17, 2023 6:40 pm UTC. (2023). “Cisco buried the lede.” >10,000 network devices backdoored through unpatched 0-day. Retrieved from https://arstechnica.com/security/2023/10/actively-exploited-cisco-0-day-with-maximum-10-severity-gives-full-network-control/?utm_social-type=owned&utm_brand=ars&utm_medium=social&utm_source=twitter