data not visible in Kibana - softflowd generated Netflow v9 - python script shows flow data
Hi , I did a fresh install on Ubuntu. I am using softflowd to generate Netflow data for testing. The data shows up on the Python Netflow v9 script debug logs, but I am not able to see anything on Kibana. Not sure if data is getting to ES. Here is the sample o/p from Python script. Looking at this log, Python script seems to do everything fine. I am running everything on the same system.
Kibana shows "no data" for index "flow*" .
thx for your help.
INFO:root:Data flow number 9, set ID 1024 from 127.0.0.1 DEBUG:root:{'_type': 'Flow', '_source': {'Protocol Number': 6, 'Last Switched': 3205735, 'Source Domain': '104.210.4.191', 'Content': 'Uncategorized', 'IPv4 Destination': '192.168.1.102', 'Sensor': '127.0.0.1', 'Bytes In': 524, 'Flow Type': 'Netflow v9', 'Input Interface': 0, 'TCP Flags': 27, 'Traffic': 'HTTPS', 'IPv4 Source': '104.210.4.191', 'Destination Port': 50776, 'IP Protocol Version': 4, 'Traffic Category': 'Web', 'Destination Domain': '192.168.1.102', 'Time': '2017-08-17T05:42:01.846Z', 'Destination FQDN': '192.168.1.102', 'Protocol': 'TCP', 'Sequence': 433, 'Source FQDN': '104.210.4.191', 'Output Interface': 0, 'Source Port': 443, 'Packets In': 7, 'Source ID': 0, 'First Switched': 3199421}, '_index': 'flow-2017-08-17'} INFO:root:Ending data flow 9 INFO:root:Data flow number 10, set ID 1024 from 127.0.0.1 DEBUG:root:{'_type': 'Flow', '_source': {'Protocol Number': 6, 'Last Switched': 3205735, 'Source Domain': '192.168.1.102', 'Content': 'Uncategorized', 'IPv4 Destination': '104.210.4.191', 'Sensor': '127.0.0.1', 'Bytes In': 1023, 'Flow Type': 'Netflow v9', 'Input Interface': 0, 'TCP Flags': 27, 'Traffic': 'HTTPS', 'IPv4 Source': '192.168.1.102', 'Destination Port': 443, 'IP Protocol Version': 4, 'Traffic Category': 'Web', 'Destination Domain': '104.210.4.191', 'Time': '2017-08-17T05:42:01.846Z', 'Destination FQDN': '104.210.4.191', 'Protocol': 'TCP', 'Sequence': 433, 'Source FQDN': '192.168.1.102', 'Output Interface': 0, 'Source Port': 50776, 'Packets In': 8, 'Source ID': 0, 'First Switched': 3199421}, '_index': 'flow-2017-08-17'} INFO:root:Ending data flow 10 INFO:root:Finished set 1024, position 416 INFO:root:Out of bytes to unpack, stopping - OK DEBUG:root:Cached templates: {6696883770888833947: {'Type': 'Flow Data', 'Template ID': 1024, 'Sensor': '127.0.0.1', 'Length': 13, 'Definitions': OrderedDict([(8, 4), (12, 4), (21, 4), (22, 4), (1, 4), (2, 4), (10, 4), (14, 4), (7, 2), (11, 2), (4, 1), (6, 1), (60, 1)])}}