Understanding the Differences Between Flow Logs on AWS and Azure
Summary
AWS VPC flow Logs and Azure NSG flow Logs offer network traffic visibility with different scopes and formats, but both are essential for multi-cloud network management and security. Unified network observability solutions analyze both in one place to provide comprehensive insights across clouds.
Visibility into our AWS and Azure cloud environments requires collecting AWS VPC flow logs and Azure NSG flow logs. These logs provide a granular view of network traffic, offering insights into connections, bandwidth usage, security incidents, and more. And while these two services are functionally similar, their implementation, features, format, and use cases are different.
AWS VPC flow logs
AWS VPC flow logs capture information about the IP traffic going to and from network interfaces within a VPC. They’re typically used to monitor and troubleshoot network connectivity and ensure compliance with security policies.
VPC flow logs operate at the VPC level, subnet level, or individual ENI (elastic network interface) level. This means engineers can use VPC flow logs to capture traffic across multiple layers of the AWS network hierarchy. Logs capture inbound and outbound traffic at the packet level so you can get information about packet and byte count, source and destination addresses, ports, and protocols.
This is in contrast to Azure NSG flow logs, which monitor the traffic filtered by the NSG security rules. That means Azure VPC flow logs will be more broad in scope and more granular in depth.
VPC flow logs are stored in Amazon CloudWatch Logs (if you’re using CloudWatch) or Amazon S3, which allows for relatively easy integration with other AWS services like Athena for analytics or AWS Glue for data processing.
The structure of VPC flow logs is interesting. They’re text-based and consist of useful fields such as:
- Version
- Account ID
- Interface ID
- Source and destination IP
- Protocol
- Source and destination port
- Action (ACCEPT/REJECT)
- Log status (OK, NODATA)
We can use these logs for a variety of cases beyond mapping traffic by source, destination, port, and protocol. For example, we can use them to identify connectivity issues between EC2 instances, map latency and bottlenecks, detect unauthorized access attempts to verify our traffic adheres to regulatory compliance, and so on. However, one thing to note is that VPC flow logs are somewhat static and aren’t highly extensible beyond this.
Below you can see an example of a raw flow record from the AWS website showing SSH traffic from 172.31.16.139 to a network interface with private IP address is 172.31.16.21 and ID eni-1235b8ca123456789 in account 123456789010 was allowed.
2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK
Azure NSG flow logs
Azure Network Security Group (NSG) flow Logs monitor traffic flowing through NSGs, which control access to Azure virtual networks. NSG flow logs provide layer 4 traffic visibility and (naturally) integrate tightly with Azure’s security and analytics ecosystem.
NSG flow Logs operate at the NSG level, which means they capture traffic for resources protected by the network security group, such as VMs and subnets. Therefore, use cases for using NSG flow logs often have to do with security. Additionally, like VPC flow logs, NSG flow Logs offer enriched data, but they deliver it in JSON format, another crucial difference between the two log types.
In this output below from Azure’s website, we can see a version 2 NSG flow record in a structured JSON format:
{
"records": [
{
"time": "2018-11-13T12:00:35.3899262Z",
"systemId": "66aa66aa-bb77-cc88-dd99-00ee00ee00ee",
"category": "NetworkSecurityGroupFlowEvent",
"resourceId": "/SUBSCRIPTIONS/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG",
"operationName": "NetworkSecurityGroupFlowEvents",
"properties": {
"Version": 2,
"flows": [
{
"rule": "DefaultRule\_DenyAllInBound",
"flows": [
{
"mac": "000D3AF87856",
"flowTuples": [
"1542110402,192.0.2.190,10.5.16.4,28746,443,U,I,D,B,,,,",
"1542110424,203.0.113.10,10.5.16.4,56509,59336,T,I,D,B,,,,",
"1542110432,198.51.100.8,10.5.16.4,48495,8088,T,I,D,B,,,,"
]
}
]
},
{
"rule": "DefaultRule\_AllowInternetOutBound",
"flows": [
{
"mac": "000D3AF87856",
"flowTuples": [
"1542110377,10.5.16.4,203.0.113.118,59831,443,T,O,A,B,,,,",
"1542110379,10.5.16.4,203.0.113.117,59932,443,T,O,A,E,1,66,1,66",
"1542110379,10.5.16.4,203.0.113.115,44931,443,T,O,A,C,30,16978,24,14008",
"1542110406,10.5.16.4,198.51.100.225,59929,443,T,O,A,E,15,8489,12,7054"
]
}
]
}
]
}
},
{
"time": "2018-11-13T12:01:35.3918317Z",
"systemId": "66aa66aa-bb77-cc88-dd99-00ee00ee00ee",
"category": "NetworkSecurityGroupFlowEvent",
"resourceId": "/SUBSCRIPTIONS/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG",
"operationName": "NetworkSecurityGroupFlowEvents",
"properties": {
"Version": 2,
"flows": [
{
"rule": "DefaultRule\_DenyAllInBound",
"flows": [
{
"mac": "000D3AF87856",
"flowTuples": [
"1542110437,125.64.94.197,10.5.16.4,59752,18264,T,I,D,B,,,,",
"1542110475,80.211.72.221,10.5.16.4,37433,8088,T,I,D,B,,,,",
"1542110487,46.101.199.124,10.5.16.4,60577,8088,T,I,D,B,,,,",
"1542110490,176.119.4.30,10.5.16.4,57067,52801,T,I,D,B,,,,"
]
}
]
}
]
}
}
]
As you can see above, some of the fields in an NSG flow log include:
- Source and destination IP
- Source and destination port
- Protocol
- Traffic flow direction (Inbound/Outbound)
- Traffic flow state
- Rule name (denoting the NSG rule responsible for allowing or denying traffic)
This may seem very similar to AWS VPC flow logs at first, but an important distinction is that the structured format of NSG flow logs is extensible and works well for both automation and advanced analytics. That, along with the focus on NSG flow logs on the actual security rules protecting subnets, means we can use NSG flow logs for more than just source and destination traffic analysis - can use them for more advanced security use cases such as detecting rule misconfigurations or potential attacks, understanding traffic patterns to optimize routing and security rules, and even for forensic analysis such as investigating incidents by analyzing historical flows.
Azure utilizes its own set of tools to store and analyze NSG flow logs. Logs are stored in Azure Storage Accounts and can be analyzed with Azure Monitor, Azure Sentinel, or external tools like Splunk.
The right tool for the job
Visibility into cloud environments is critical for effective network management, security, and optimization. AWS VPC flow logs and Azure NSG flow logs offer invaluable insights into network traffic. Still, their differences in structure, capabilities, and integration with their respective ecosystems highlight the importance of choosing the right tool for your specific needs.
In today’s multi-cloud environments, where AWS and Azure often coexist, organizations require a unified observability solution capable of seamlessly handling both log formats. That way, teams can centralize data analysis, correlate traffic patterns, and gain comprehensive visibility across cloud platforms.
