NSX-T IDS/IPS Event forwarding to Splunk – Part 1

The intrusion detection and prevention system (IDS/IPS) functionality released with NSX-T 3.0 provides great visibility and control of attacks in your SDDC, but these can only be consumed out-of-the-box from within the NSX-T product itself. As an Enterprise customer, I would like these events to be forwarded to a SIEM – in my case this is Splunk.

This article is written in two parts:

• Part 1: – Setting up IDS/IPS alerting to vRealize Log Insight
• Part 2: – Forwarding events to Splunk and viewing them

Products tested against are:

• NSX-T 3.1.3
• ESXi 7.0u3
• Splunk Enterprise 8.2.2.1

Update 22.01.2022 – There is a change in the appname field introduced in NSX 3.2.0. IDS/IPS logs are now labelled with the appname IDPS-EVT instead of IDPS_EVT.

You probably have this already configured if you have deployed vRLI, but it is worth a check:

esxcli system syslog config get

You might notice that the default value of Log Level in esxcli is “error”. Don’t worry (or change it), the IDS/IPS events are sent to the syslog with this value.

Empiric testing of setting Log Level to “info” in the esxcli has had no discernible difference to “error” – in vRLI I saw 16.3 K logs when esxcli was set to “info” over 5 minutes, and 17K logs when set to “error” over 5 minutes. There is no difference between the types of logged events either.

Hence, if it ain’t broke, don’t fix it!

Note: There currently is a “loose” correlation between the Advanced System Settings in the vSphere Web Client and the esxcli command. For example, if you change Syslog.global.logHost in the GUI it will reflect as “Remote Host” on the esxcli, but there are some parameters that are only in the GUI and others that are only in the CLI (more on that later).

If logs are not being received by vRLI (or Splunk) use this command to test if the port is reachable from the ESXi host:

nc -z RemoteHostname 514

There are some older articles on the Internet that state the need to perform a “set ids engine syslogstatus enable” command in the nsxcli on each ESXi host. This command has been deprecated as it did not survive an ESXi reboot. The only way to do this now is via the NSX Manager API.

1. First GET the IDS-GlobalConfig:
   GET: https:///api/v1/global-configs/IdsGlobalConfig
   Response Example:

  {
  "global_idsevents_to_syslog_enabled": false,
  "resource_type": "IdsGlobalConfig",
  "id": "ad4f0184-e92a-436a-bfb6-402e815b6163",
  "display_name": "ad4f0184-e92a-436a-bfb6-402e815b6163",
  "_create_user": "system",
  "_create_time": 1587552586050,
  "_last_modified_user": "admin",
  "_last_modified_time": 1618557056672,
  "_system_owned": false,
  "_protection": "NOT_PROTECTED",
  "_revision": 118
}

2. Now you can enable the usage of syslog for IDS events through this PUT command (make sure you use the same revision number that you got in the GET command above):

PUT: https:///api/v1/global-configs/IdsGlobalConfig
Body:

{
  "global_idsevents_to_syslog_enabled": true,
  "resource_type": "IdsGlobalConfig",
  "_revision": 118
}

3. If you execute the GET command in step 1 again, you will see that the revision number has increased.

1. Authenticate to vRLI and go to Interactive Analysis view
2. Search for IDPS-EVT (for NSX 3.2.0+. Earlier NSX versions use IDPS_EVT)

You may find some of the IDS events have been truncated – for example in the screenshot below, the payload has been truncated at the flow_dest_ip value.

OK, let’s look at the log sender (the ESXi host). There is a file  /scratch/log/nsx-idps/nsx-idps-events.log that sounds like a candidate. Yep, the event is here and is truncated at the same spot:

This issue is caused by the strict adherence to RFC 3164 on the vmsyslog daemon side which sets the maximum message length to 1024 bytes.

Luckily there is a workaround which was introduced in ESXi 6.7 P04 but note these recommendations:

• UDP syslog messages should not exceed 4096 bytes
• TCP syslog messages can be increased to 16,384 bytes

esxcli system syslog config set --remote-host-max-msg-len=16384

After making configuration changes, load the new configuration by executing

esxcli system syslog reload

Here is a script that will help you to set the maximum message length value in bulk on a per-cluster basis, with a CSV file output to verify what has changed:

<#
.SYNOPSIS Script to set the maximum message length for ESXi syslog
.NOTES  Author:  Lucas Bernadsky
.NOTES  Site:    www.cloudblog.tech
.NOTES Thanks to virten.net for his helpful article about ESXCLI v2. https://www.virten.net/2016/11/how-to-use-esxcli-v2-commands-in-powercli/

.VERSION 1.0 - 04.11.2021

.EXAMPLE PS> ./max_remote_msg_length.ps1
#>


#region Connect to vCenter

# Clear existing vCenter connections
$global:DefaultVIServers = $null

# Validate vCenter connection
do{
    Write-Host "`nPlease connect to vCenter" -ForegroundColor Green
	Connect-VIServer -ErrorAction SilentlyContinue -Verbose:$False
    
    if ($global:DefaultVIServers.Count -eq 0){
        Write-Host "There has been a problem connecting to a vCenter Server. Please try again." -ForegroundColor Red
    }

}while($global:DefaultVIServers.Count -eq 0)

#endregion

#region Select cluster
$Cluster = $null
do{
    Write-Host "`nClusters list" -ForegroundColor Yellow
    Get-Cluster | Select name | Sort-Object -Property Name 
    $Cluster = Read-Host 'Please enter cluster name'
    $Cluster = Get-Cluster -Name $Cluster -ErrorAction SilentlyContinue
    
    if($Cluster -ne $null){
    
        write-Host $Cluster.Name
    }else{
    
    $Cluster
    write-host 'Cluster not found. Please try again' -ForegroundColor Red
    
    }
}while($Cluster -eq $null)

Write-Host $Cluster.Name " selected." -ForegroundColor Green
#endregion


#region Confirm operation
$Confirmation = $null
do{

    $Confirmation = Read-Host "Are you sure you want to proceed with the changes? (yes/no)"

}while($Confirmation -ne "y" -and $Confirmation -ne "yes" -and $Confirmation -ne "n" -and $Confirmation -ne "no")

if($Confirmation -eq "n" -or $Confirmation -eq "no"){
	# Disconnects all vCenter sessions.
    Write-Host "Disconnecting" -ForegroundColor Yellow
    Disconnect-VIServer * -Confirm:$false
    
	# Terminate script
	break;
}
#endregion

#region Change remote host maximum message length
$Nodes = $Cluster | Get-VMhost

$output_summary = @()


foreach ($esxi in $Nodes){
	
	write-Host "Changing parameters on host " $esxi.name -ForegroundColor Yellow
	$Output_Object = New-Object -TypeName PSObject
	
    $esxcli = Get-EsxCli -V2 -VMHost $esxi
	#$esxcli.system.syslog.config.get.Invoke()
    $arguments = $esxcli.system.syslog.config.set.CreateArgs() 
    
    #List arguments $arguments

    $arguments.remotehostmaxmsglen = 16384
    $esxcli.system.syslog.config.set.Invoke($arguments) | Out-Null
    $esxcli.system.syslog.reload.Invoke() | Out-Null

#region Populate Output Object   
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name esxi_name -Value $esxi.name.ToString()
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name CheckCertificateRevocation -Value $esxcli.system.syslog.config.get.Invoke().CheckCertificateRevocation
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name DefaultNetworkRetryTimeout -Value $esxcli.system.syslog.config.get.Invoke().DefaultNetworkRetryTimeout
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name DroppedLogFileRotationSize -Value $esxcli.system.syslog.config.get.Invoke().DroppedLogFileRotationSize
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name DroppedLogFileRotations -Value $esxcli.system.syslog.config.get.Invoke().DroppedLogFileRotations
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name EnforceSSLCertificates -Value $esxcli.system.syslog.config.get.Invoke().EnforceSSLCertificates
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name LocalLogOutput -Value $esxcli.system.syslog.config.get.Invoke().LocalLogOutput
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name LocalLogOutputIsConfigured -Value $esxcli.system.syslog.config.get.Invoke().LocalLogOutputIsConfigured
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name LocalLogOutputIsPersistent -Value $esxcli.system.syslog.config.get.Invoke().LocalLogOutputIsPersistent
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name LocalLoggingDefaultRotationSize -Value $esxcli.system.syslog.config.get.Invoke().LocalLoggingDefaultRotationSize
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name LocalLoggingDefaultRotations -Value $esxcli.system.syslog.config.get.Invoke().LocalLoggingDefaultRotations
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name LogLevel -Value $esxcli.system.syslog.config.get.Invoke().LogLevel
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name LogToUniqueSubdirectory -Value $esxcli.system.syslog.config.get.Invoke().LogToUniqueSubdirectory
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name MessageQueueDropMark -Value $esxcli.system.syslog.config.get.Invoke().MessageQueueDropMark
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name RemoteHost -Value $esxcli.system.syslog.config.get.Invoke().RemoteHost
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name RemoteHostMaximumMessageLength -Value $esxcli.system.syslog.config.get.Invoke().RemoteHostMaximumMessageLength
	Add-Member -InputObject $Output_Object -MemberType NoteProperty -Name StrictX509Compliance -Value $esxcli.system.syslog.config.get.Invoke().StrictX509Compliance

	$output_summary += $Output_Object
#endregion	
}
$output_summary | Export-Csv .\max_remote_msg_length.csv -NoTypeInformation

$location = Get-location
Write-Host "Please find log file in " $location"\max_remote_msg_length.csv"  -ForegroundColor Green

#endregion

# Disconnects all vCenter sessions.
Disconnect-VIServer * -Confirm:$false

After changing this value on an ESXi host, we can see that vRLI (and also Splunk) has received the full event payload. Some IDPS_EVT payloads are smaller – this behemoth is the largest single event found in my lab: