Skip to content
BranchesTags

Repository files navigation

NTLFlowLyzer

As part of the Understanding Cybersecurity Series (UCS), NTLFlowLyzer is a Python open-source project to extract network layer features from TCP-based network traffic for Anomaly Profiling (AP) which is the second component of the NetFlowLyzer.

NTLFlowLyzer generates bidirectional flows from the Network and Transportation Layers of network traffic, where the first packet determines the forward (source to destination) and backward (destination to source) directions, hence the statistical time-related features can be calculated separately in the forward and backward directions. Additional functionalities include selecting features from the list of existing features, adding new features, and controlling the duration of flow timeout. Moreover, TCP flows are terminated upon connection teardown (by FIN or RST packet), reaching the flow's maximum duration, or being inactive for a certain amount of time (timeout).

Table of Contents

Installation

Before installing or running the NTLFlowLyzer package, it's essential to set up the necessary requirements on your system. Begin by ensuring you have both Python and pip installed and functioning properly (execute the pip3 --version command). Then, execute the following command:

pip3 install -r requirements.txt

You are prepared to install NTLFlowLyzer. To proceed, execute the following command in the package's root directory (where the setup.py file is located), which will install the NTLFlowLyzer package on your system:

On Linux:

python3 setup.py install

On Windows:

pip3 install .

After successfully installing the package, confirm the installation by running the following command:

ntlflowlyzer --version

Execution

The core aspect of running NTLFlowLyzer involves preparing the configuration file. This file is designed to facilitate users in customizing the program's behavior with minimal complexity and cost, thus enhancing program scalability. Below, we outline how to prepare the configuration file and subsequently demonstrate how to execute NTLFlowLyzer using it.

Configuration File

The configuration file is formatted in JSON, comprising key-value pairs that enable customization of the package. While some keys are mandatory, others are optional. Below, each key is explained along with its corresponding value:

  • pcap_file_address [Required]

    This key specifies the input PCAP file address. The format of the value should be a string.

    Note: At this version of NTLFlowLyzer, we only support the PCAP format. For other formats such as PCAPNG, you must convert them to PCAP. To convert PCAPNG to PCAP, you can use Wireshark. If you prefer command-line tools, you can use the following command:

    tshark -F pcap -r {pcapng_file} -w {pcap_file}

    Replace {pcapng_file} with the path to your PCAPNG file and {pcap_file} with the desired output PCAP file name.

  • output_file_address [Required]

    This key specifies the output CSV file address. The format of the value should be a string.

  • label [Optional]

    This key specifies the value of the label column in the output CSV file address. The format of the value should be a string. The default value is Unknown.

  • number_of_threads [Optional]

    This key specifies the number of threads to be used for all processes, including flow extraction, feature calculation, and output writing. The value must be an integer of at least 3. The default value is 4.

    It's important to consider that the optimal value for this option varies based on the system configuration and the format of the input PCAP file. For instance, if the PCAP file contains a large number of packets (e.g., more than 5 million) and they are all TCP packets, increasing the number of threads might be beneficial. However, if the packets represent a small number of flows and all related packets are contiguous, adding more threads could potentially slow down the program since there are fewer distinct flows.

    As a rule of thumb, the ideal value for this option typically falls between half the number of CPU cores (CPU count) and twice the CPU count. This helps balance computational resources without overwhelming the system. (0.5 * cpu_count < best_option < 2 * cpu_count)

  • feature_extractor_min_flows [Optional]

    This key determines the minimum number of finished flows required for the feature extractor thread to initiate its work and extract features from these finished flows. The value must be an integer. The default value is 4000.

    Selecting a high value for this option will consume more RAM since more flows will be stored in memory, potentially slowing down the entire program. Conversely, choosing a low value for this option can slow down the execution process, as it involves locking the finished flows list and then copying those flows for feature extraction. These two processes, locking and copying, are slow and can impede other program components.

  • writer_min_rows [Optional]

    This key specifies the minimum number of ready flows (i.e., finished flows from which features have been extracted) required for the writer thread to begin its work of writing the flows to the CSV file. The value must be an integer. The default value is 6000.

    Opting for a high value for this option will increase RAM usage since more flows will be stored in memory, potentially slowing down the overall program performance. Conversely, selecting a low value for this option can slow down the execution process, involving locking the finished flows list, copying those flows for the writing process, and performing I/O operations to write to the file. These three processes — locking, copying, and I/O — are slow and may impede other program components.

  • read_packets_count_value_log_info [Optional]

    This key determines the minimum number of processed packets (i.e., the number of packets read from the PCAP file and assigned to a flow) required for the logger to log. The value must be an integer. The default value is 10,000. This means that after processing every 10,000 packets, the program will print a statement indicating the number of packets analyzed.

  • check_flows_ending_min_flows [Optional]

    This key specifies the minimum number of ongoing flows (i.e., created flows that have not yet finished) required for checking if they have reached the timeout or maximum flow time value. The value must be an integer. The default value is 2000. This indicates that if the number of ongoing flows exceeds 2000, the program will proceed to check all flows for timeout or maximum flow time.

  • capturer_updating_flows_min_value [Optional]

    This key determines the minimum number of finished flows required to be added to the queue for feature extraction. The value must be an integer. The default value is 2000. This means that if the number of finished flows exceeds 2000, the program will move them to a separate list for the feature extractor.

  • max_flow_duration [Optional]

    This key sets the maximum duration of a flow in seconds. The value must be an integer. The default value is 120,000. It means if the flow duration exceeds 120,000 seconds, the program will terminate the flow and initiate a new one.

  • activity_timeout [Optional]

    This key defines the flow activity timeout in seconds. The value must be an integer. The default value is 5000. It means if 5000 seconds have elapsed since the last packet of the flow, the program will terminate the flow.

  • floating_point_unit [Optional]

    This key specifies the floating point unit used for the feature extraction process. The value must be in the format: .[UNIT]f. The default value is .4f. This indicates that the feature values will be rounded to the fourth decimal place.

  • max_rows_number [Optional]

    This key defines the maximum number of rows in the output CSV file. The value must be an integer. The default value is 900,000. It means if there are more than 900,000 flows to be written in the CSV file, the program will close the current CSV file and create a new one for the remaining flows.

  • features_ignore_list [Optional]

    This key specifies the features that you do not want to extract. The value must be a list of string values, where each string represents a feature name. The default value is an empty list. If you include a feature name in this list, the program will skip extracting that feature, and it will not appear in the output CSV file.

An example of a configuration file would be like this:

{
"pcap_file_address": "/mnt/c/dataset/my_pcap_file.pcap",
"output_file_address": "./output-of-my_pcap_file.csv",
"label": "Benign",
"number_of_threads": 4,
"feature_extractor_min_flows": 2500,
"writer_min_rows": 1000,
"read_packets_count_value_log_info": 1000000,
"check_flows_ending_min_flows": 20000,
"capturer_updating_flows_min_value": 5000,
"max_flow_duration": 120000,
"activity_timeout": 300,
"floating_point_unit": ".4f",
"max_rows_number": 800000,
"features_ignore_list": ["duration", "src_ip"]
}

In general, we recommend adjusting the values of the following options: number_of_threads, feature_extractor_min_flows, writer_min_rows, check_flows_ending_min_flows, and capturer_updating_flows_min_value, based on your system configuration. This is particularly important if your PCAP file is large (usually more than 4 GB with over 1 million TCP packets), to optimize program efficiency.

Argument Parser

You can use -h to see different options of the program.

To execute NTLFlowLyzer, simply run the following command:

ntlflowlyzer -c YOUR_CONFIG_FILE

Replace YOUR_CONFIG_FILE with the path to your configuration file.

Moreover, this project has been successfully tested on Ubuntu 20.04, Ubuntu 22.04, Windows 10, and Windows 11. It should work on other versions of Ubuntu OS (or even Debian OS) as long as your system has the necessary Python3 packages (you can find the required packages listed in the requirements.txt file).

Architecture

Extracted Features

We currently have 348 features that are as follows (features' explanation will be added):

  1. flow_id
  2. src_ip
  3. src_port
  4. dst_ip
  5. dst_port
  6. protocol
  7. timestamp
  8. Duration
  9. PacketsCount
  10. FwdPacketsCount
  11. BwdPacketsCount
  12. TotalPayloadBytes
  13. FwdTotalPayloadBytes
  14. BwdTotalPayloadBytes
  15. PayloadBytesMax
  16. PayloadBytesMin
  17. PayloadBytesMean
  18. PayloadBytesStd
  19. PayloadBytesVariance
  20. PayloadBytesMedian
  21. PayloadBytesSkewness
  22. PayloadBytesCov
  23. PayloadBytesMode
  24. FwdPayloadBytesMax
  25. FwdPayloadBytesMin
  26. FwdPayloadBytesMean
  27. FwdPayloadBytesStd
  28. FwdPayloadBytesVariance
  29. FwdPayloadBytesMedian
  30. FwdPayloadBytesSkewness
  31. FwdPayloadBytesCov
  32. FwdPayloadBytesMode
  33. BwdPayloadBytesMax
  34. BwdPayloadBytesMin
  35. BwdPayloadBytesMean
  36. BwdPayloadBytesStd
  37. BwdPayloadBytesVariance
  38. BwdPayloadBytesMedian
  39. BwdPayloadBytesSkewness
  40. BwdPayloadBytesCov
  41. BwdPayloadBytesMode
  42. TotalHeaderBytes
  43. MaxHeaderBytes
  44. MinHeaderBytes
  45. MeanHeaderBytes
  46. StdHeaderBytes
  47. MedianHeaderBytes
  48. SkewnessHeaderBytes
  49. CoVHeaderBytes
  50. ModeHeaderBytes
  51. VarianceHeaderBytes
  52. FwdTotalHeaderBytes
  53. FwdMaxHeaderBytes
  54. FwdMinHeaderBytes
  55. FwdMeanHeaderBytes
  56. FwdStdHeaderBytes
  57. FwdMedianHeaderBytes
  58. FwdSkewnessHeaderBytes
  59. FwdCoVHeaderBytes
  60. FwdModeHeaderBytes
  61. FwdVarianceHeaderBytes
  62. BwdTotalHeaderBytes
  63. BwdMaxHeaderBytes
  64. BwdMinHeaderBytes
  65. BwdMeanHeaderBytes
  66. BwdStdHeaderBytes
  67. BwdMedianHeaderBytes
  68. BwdSkewnessHeaderBytes
  69. BwdCoVHeaderBytes
  70. BwdModeHeaderBytes
  71. BwdVarianceHeaderBytes
  72. FwdSegmentSizeMean
  73. FwdSegmentSizeMax
  74. FwdSegmentSizeMin
  75. FwdSegmentSizeStd
  76. FwdSegmentSizeVariance
  77. FwdSegmentSizeMedian
  78. FwdSegmentSizeSkewness
  79. FwdSegmentSizeCov
  80. FwdSegmentSizeMode
  81. BwdSegmentSizeMean
  82. BwdSegmentSizeMax
  83. BwdSegmentSizeMin
  84. BwdSegmentSizeStd
  85. BwdSegmentSizeVariance
  86. BwdSegmentSizeMedian
  87. BwdSegmentSizeSkewness
  88. BwdSegmentSizeCov
  89. BwdSegmentSizeMode
  90. SegmentSizeMean
  91. SegmentSizeMax
  92. SegmentSizeMin
  93. SegmentSizeStd
  94. SegmentSizeVariance
  95. SegmentSizeMedian
  96. SegmentSizeSkewness
  97. SegmentSizeCov
  98. SegmentSizeMode
  99. FwdInitWinBytes
  100. BwdInitWinBytes
  101. ActiveMin
  102. ActiveMax
  103. ActiveMean
  104. ActiveStd
  105. ActiveMedian
  106. ActiveSkewness
  107. ActiveCoV
  108. ActiveMode
  109. ActiveVariance
  110. IdleMin
  111. IdleMax
  112. IdleMean
  113. IdleStd
  114. IdleMedian
  115. IdleSkewness
  116. IdleCoV
  117. IdleMode
  118. IdleVariance
  119. BytesRate
  120. FwdBytesRate
  121. BwdBytesRate
  122. PacketsRate
  123. BwdPacketsRate
  124. FwdPacketsRate
  125. DownUpRate
  126. AvgFwdBytesPerBulk
  127. AvgFwdPacketsPerBulk
  128. AvgFwdBulkRate
  129. AvgBwdBytesPerBulk
  130. AvgBwdPacketsPerBulk
  131. AvgBwdBulkRate
  132. FwdBulkStateCount
  133. FwdBulkSizeTotal
  134. FwdBulkPacketCount
  135. FwdBulkDuration
  136. BwdBulkStateCount
  137. BwdBulkSizeTotal
  138. BwdBulkPacketCount
  139. BwdBulkDuration
  140. FINFlagCounts
  141. PSHFlagCounts
  142. URGFlagCounts
  143. ECEFlagCounts
  144. SYNFlagCounts
  145. ACKFlagCounts
  146. CWRFlagCounts
  147. RSTFlagCounts
  148. FwdFINFlagCounts
  149. FwdPSHFlagCounts
  150. FwdURGFlagCounts
  151. FwdECEFlagCounts
  152. FwdSYNFlagCounts
  153. FwdACKFlagCounts
  154. FwdCWRFlagCounts
  155. FwdRSTFlagCounts
  156. BwdFINFlagCounts
  157. BwdPSHFlagCounts
  158. BwdURGFlagCounts
  159. BwdECEFlagCounts
  160. BwdSYNFlagCounts
  161. BwdACKFlagCounts
  162. BwdCWRFlagCounts
  163. BwdRSTFlagCounts
  164. FINFlagPercentageInTotal
  165. PSHFlagPercentageInTotal
  166. URGFlagPercentageInTotal
  167. ECEFlagPercentageInTotal
  168. SYNFlagPercentageInTotal
  169. ACKFlagPercentageInTotal
  170. CWRFlagPercentageInTotal
  171. RSTFlagPercentageInTotal
  172. FwdFINFlagPercentageInTotal
  173. FwdPSHFlagPercentageInTotal
  174. FwdURGFlagPercentageInTotal
  175. FwdECEFlagPercentageInTotal
  176. FwdSYNFlagPercentageInTotal
  177. FwdACKFlagPercentageInTotal
  178. FwdCWRFlagPercentageInTotal
  179. FwdRSTFlagPercentageInTotal
  180. BwdFINFlagPercentageInTotal
  181. BwdPSHFlagPercentageInTotal
  182. BwdURGFlagPercentageInTotal
  183. BwdECEFlagPercentageInTotal
  184. BwdSYNFlagPercentageInTotal
  185. BwdACKFlagPercentageInTotal
  186. BwdCWRFlagPercentageInTotal
  187. BwdRSTFlagPercentageInTotal
  188. FwdFINFlagPercentageInFwdPackets
  189. FwdPSHFlagPercentageInFwdPackets
  190. FwdURGFlagPercentageInFwdPackets
  191. FwdECEFlagPercentageInFwdPackets
  192. FwdSYNFlagPercentageInFwdPackets
  193. FwdACKFlagPercentageInFwdPackets
  194. FwdCWRFlagPercentageInFwdPackets
  195. FwdRSTFlagPercentageInFwdPackets
  196. BwdFINFlagPercentageInBwdPackets
  197. BwdPSHFlagPercentageInBwdPackets
  198. BwdURGFlagPercentageInBwdPackets
  199. BwdECEFlagPercentageInBwdPackets
  200. BwdSYNFlagPercentageInBwdPackets
  201. BwdACKFlagPercentageInBwdPackets
  202. BwdCWRFlagPercentageInBwdPackets
  203. BwdRSTFlagPercentageInBwdPackets
  204. PacketsIATMean
  205. PacketsIATStd
  206. PacketsIATMax
  207. PacketsIATMin
  208. PacketsIATSum
  209. PacketsIATMedian
  210. PacketsIATSkewness
  211. PacketsIATCoV
  212. PacketsIATMode
  213. PacketsIATVariance
  214. FwdPacketsIATMean
  215. FwdPacketsIATStd
  216. FwdPacketsIATMax
  217. FwdPacketsIATMin
  218. FwdPacketsIATSum
  219. FwdPacketsIATMedian
  220. FwdPacketsIATSkewness
  221. FwdPacketsIATCoV
  222. FwdPacketsIATMode
  223. FwdPacketsIATVariance
  224. BwdPacketsIATMean
  225. BwdPacketsIATStd
  226. BwdPacketsIATMax
  227. BwdPacketsIATMin
  228. BwdPacketsIATSum
  229. BwdPacketsIATMedian
  230. BwdPacketsIATSkewness
  231. BwdPacketsIATCoV
  232. BwdPacketsIATMode
  233. BwdPacketsIATVariance
  234. SubflowFwdPackets
  235. SubflowBwdPackets
  236. SubflowFwdBytes
  237. SubflowBwdBytes
  238. DeltaStart
  239. HandshakeDuration
  240. HandshakeState
  241. PacketsDeltaTimeMin
  242. PacketsDeltaTimeMax
  243. PacketsDeltaTimeMean
  244. PacketsDeltaTimeMode
  245. PacketsDeltaTimeVariance
  246. PacketsDeltaTimeStd
  247. PacketsDeltaTimeMedian
  248. PacketsDeltaTimeSkewness
  249. PacketsDeltaTimeCoV
  250. BwdPacketsDeltaTimeMin
  251. BwdPacketsDeltaTimeMax
  252. BwdPacketsDeltaTimeMean
  253. BwdPacketsDeltaTimeMode
  254. BwdPacketsDeltaTimeVariance
  255. BwdPacketsDeltaTimeStd
  256. BwdPacketsDeltaTimeMedian
  257. BwdPacketsDeltaTimeSkewness
  258. BwdPacketsDeltaTimeCoV
  259. FwdPacketsDeltaTimeMin
  260. FwdPacketsDeltaTimeMax
  261. FwdPacketsDeltaTimeMean
  262. FwdPacketsDeltaTimeMode
  263. FwdPacketsDeltaTimeVariance
  264. FwdPacketsDeltaTimeStd
  265. FwdPacketsDeltaTimeMedian
  266. FwdPacketsDeltaTimeSkewness
  267. FwdPacketsDeltaTimeCoV
  268. PacketsDeltaLenMin
  269. PacketsDeltaLenMax
  270. PacketsDeltaLenMean
  271. PacketsDeltaLenMode
  272. PacketsDeltaLenVariance
  273. PacketsDeltaLenStd
  274. PacketsDeltaLenMedian
  275. PacketsDeltaLenSkewness
  276. PacketsDeltaLenCoV
  277. BwdPacketsDeltaLenMin
  278. BwdPacketsDeltaLenMax
  279. BwdPacketsDeltaLenMean
  280. BwdPacketsDeltaLenMode
  281. BwdPacketsDeltaLenVariance
  282. BwdPacketsDeltaLenStd
  283. BwdPacketsDeltaLenMedian
  284. BwdPacketsDeltaLenSkewness
  285. BwdPacketsDeltaLenCoV
  286. FwdPacketsDeltaLenMin
  287. FwdPacketsDeltaLenMax
  288. FwdPacketsDeltaLenMean
  289. FwdPacketsDeltaLenMode
  290. FwdPacketsDeltaLenVariance
  291. FwdPacketsDeltaLenStd
  292. FwdPacketsDeltaLenMedian
  293. FwdPacketsDeltaLenSkewness
  294. FwdPacketsDeltaLenCoV
  295. HeaderBytesDeltaLenMin
  296. HeaderBytesDeltaLenMax
  297. HeaderBytesDeltaLenMean
  298. HeaderBytesDeltaLenMode
  299. HeaderBytesDeltaLenVariance
  300. HeaderBytesDeltaLenStd
  301. HeaderBytesDeltaLenMedian
  302. HeaderBytesDeltaLenSkewness
  303. HeaderBytesDeltaLenCoV
  304. BwdHeaderBytesDeltaLenMin
  305. BwdHeaderBytesDeltaLenMax
  306. BwdHeaderBytesDeltaLenMean
  307. BwdHeaderBytesDeltaLenMode
  308. BwdHeaderBytesDeltaLenVariance
  309. BwdHeaderBytesDeltaLenStd
  310. BwdHeaderBytesDeltaLenMedian
  311. BwdHeaderBytesDeltaLenSkewness
  312. BwdHeaderBytesDeltaLenCoV
  313. FwdHeaderBytesDeltaLenMin
  314. FwdHeaderBytesDeltaLenMax
  315. FwdHeaderBytesDeltaLenMean
  316. FwdHeaderBytesDeltaLenMode
  317. FwdHeaderBytesDeltaLenVariance
  318. FwdHeaderBytesDeltaLenStd
  319. FwdHeaderBytesDeltaLenMedian
  320. FwdHeaderBytesDeltaLenSkewness
  321. FwdHeaderBytesDeltaLenCoV
  322. PayloadBytesDeltaLenMin
  323. PayloadBytesDeltaLenMax
  324. PayloadBytesDeltaLenMean
  325. PayloadBytesDeltaLenMode
  326. PayloadBytesDeltaLenVariance
  327. PayloadBytesDeltaLenStd
  328. PayloadBytesDeltaLenMedian
  329. PayloadBytesDeltaLenSkewness
  330. PayloadBytesDeltaLenCoV
  331. BwdPayloadBytesDeltaLenMin
  332. BwdPayloadBytesDeltaLenMax
  333. BwdPayloadBytesDeltaLenMean
  334. BwdPayloadBytesDeltaLenMode
  335. BwdPayloadBytesDeltaLenVariance
  336. BwdPayloadBytesDeltaLenStd
  337. BwdPayloadBytesDeltaLenMedian
  338. BwdPayloadBytesDeltaLenSkewness
  339. BwdPayloadBytesDeltaLenCoV
  340. FwdPayloadBytesDeltaLenMin
  341. FwdPayloadBytesDeltaLenMax
  342. FwdPayloadBytesDeltaLenMean
  343. FwdPayloadBytesDeltaLenMode
  344. FwdPayloadBytesDeltaLenVariance
  345. FwdPayloadBytesDeltaLenStd
  346. FwdPayloadBytesDeltaLenMedian
  347. FwdPayloadBytesDeltaLenSkewness
  348. FwdPayloadBytesDeltaLenCoV

Definitions

  • IAT
  • Bulk
  • Subflow
  • Idle

Statistical Information Calculation

We use differnet libraries to calculate various mathematical equations. Below you can see the libraries and their brief definition based on their documentations:

  • statistics

    This module provides functions for calculating mathematical statistics of numeric (Real-valued) data.

    The module is not intended to be a competitor to third-party libraries such as NumPy, SciPy, or proprietary full-featured statistics packages aimed at professional statisticians such as Minitab, SAS and Matlab. It is aimed at the level of graphing and scientific calculators.

Nine mathematical functions are used to extract different features. You can see how those functions are calculated in the NTLFlowLyzer below:

  1. Min

    You know what it means :). The 'min' function (Python built-in) calculates the minimum value in a given list.

  2. Max

    Same as min. The 'max' function (Python built-in) calculates the minimum value in a given list.

  3. Mean

    The 'mean' function from 'statistics' library (Python built-in) calculates the mean value of a given list. According to the library documentation:

    The arithmetic mean is the sum of the data divided by the number of data points. It is commonly called “the average”, although it is only one of many different mathematical averages. It is a measure of the central location of the data.

    This runs faster than the mean() function and it always returns a float. The data may be a sequence or iterable. If the input dataset is empty, raises a StatisticsError.

  4. Standard Deviation

    The 'pstdev' function from 'statistics' library (Python built-in) calculates the mean value of a given list. According to the library documentation:

    Return the population standard deviation (the square root of the population variance). See pvariance() for arguments and other details.

Output

flow_idtimestampsrc_ipsrc_portdst_ipdst_portprotocoldurationpackets_countfwd_packets_countbwd_packets_counttotal_payload_bytesfwd_total_payload_bytesbwd_total_payload_bytespayload_bytes_maxpayload_bytes_minpayload_bytes_meanpayload_bytes_stdpayload_bytes_variancefwd_payload_bytes_maxfwd_payload_bytes_minfwd_payload_bytes_meanfwd_payload_bytes_stdfwd_payload_bytes_variancebwd_payload_bytes_maxbwd_payload_bytes_minbwd_payload_bytes_meanbwd_payload_bytes_stdbwd_payload_bytes_variancetotal_header_bytesmax_header_bytesmin_header_bytesmean_header_bytesstd_header_bytesfwd_total_header_bytesfwd_max_header_bytesfwd_min_header_bytesfwd_mean_header_bytesfwd_std_header_bytesbwd_total_header_bytesbwd_max_header_bytesbwd_min_header_bytesbwd_mean_header_bytesbwd_std_header_bytesfwd_avg_segment_sizebwd_avg_segment_sizeavg_segment_sizefwd_init_win_bytesbwd_init_win_bytesactive_minactive_maxactive_meanactive_stdidle_minidle_maxidle_meanidle_stdbytes_ratefwd_bytes_ratebwd_bytes_ratepackets_ratebwd_packets_ratefwd_packets_ratedown_up_rateavg_fwd_bytes_per_bulkavg_fwd_packets_per_bulkavg_fwd_bulk_rateavg_bwd_bytes_per_bulkavg_bwd_packets_bulk_rateavg_bwd_bulk_ratefwd_bulk_state_countfwd_bulk_total_sizefwd_bulk_per_packetfwd_bulk_durationbwd_bulk_state_countbwd_bulk_total_sizebwd_bulk_per_packetbwd_bulk_durationfin_flag_countspsh_flag_countsurg_flag_countsece_flag_countssyn_flag_countsack_flag_countscwr_flag_countsrst_flag_countsfwd_fin_flag_countsfwd_psh_flag_countsfwd_urg_flag_countsfwd_ece_flag_countsfwd_syn_flag_countsfwd_ack_flag_countsfwd_cwr_flag_countsfwd_rst_flag_countsbwd_fin_flag_countsbwd_psh_flag_countsbwd_urg_flag_countsbwd_ece_flag_countsbwd_syn_flag_countsbwd_ack_flag_countsbwd_cwr_flag_countsbwd_rst_flag_countspackets_IAT_meanpacket_IAT_stdpacket_IAT_maxpacket_IAT_minpacket_IAT_totalfwd_packets_IAT_meanfwd_packets_IAT_stdfwd_packets_IAT_maxfwd_packets_IAT_minfwd_packets_IAT_totalbwd_packets_IAT_meanbwd_packets_IAT_stdbwd_packets_IAT_maxbwd_packets_IAT_minbwd_packets_IAT_totalsubflow_fwd_packetssubflow_bwd_packetssubflow_fwd_bytessubflow_bwd_bytes
192.168.43.116_52807_94.182.113.152_443_TCP_2022-07-27 18:15:06.8519072022-07-27 14:15:06.851907192.168.43.1165280794.182.113.152443TCP35.19028516057103107851650610134514000674.0687500000000227373675443232059478759765625000000000000000000000641.5775491111246537911938503384590148925781250000000000000000000000411621.751523437502328306436538696289062500000000000000000000000000000014000674.0687500000000227373675443232059478759765625000000000000000000000641.577549111124653791193850338459014892578125000000000000000000000028619.489073561093391617760062217712402343750000000000000000000000000014000674.0687500000000227373675443232059478759765625000000000000000000000641.5775491111246537911938503384590148925781250000000000000000000000354057.94683759071631357073783874511718750000000000000000000000000000003224322020.14999999999999857891452847979962825775146484375000000000000000001.33322916259733825761202297144336625933647155761718750000000000001152322020.21052631578947256230094353668391704559326171875000000000000000001.57543468916797535506191252352437004446983337402343750000000000002072322020.11650485436893376345324213616549968719482421875000000000000000001.1766413520421838967422445421107113361358642578125000000000000000114.14035087719299983.9320388349514674.068756424064240000000003064.794729568118184.880571441805592879.9141581263124.54670941141852052.92694418360067271.61976522781784781.80701754385964921402.08.0135714.6314311988824633.2520.01305955.015971053422804160.020661498533800.075449287002159001310015600156001103000.22132254716981134001763109608873492106795310974121093750000000002.387791245475005652565414493437856435775756835937500000000000000029.9477970.035.1902850.62839794642857138562419549998594447970390319824218750000000000003.991594547989778973828833841253072023391723632812500000000000000029.9478415.7e-0535.1902850.34470762745098038060120870795799419283866882324218750000000000002.978997962197461379929563918267376720905303955078125000000000000029.9913460.035.16017799999999528.551.53253.03253.0
192.168.43.116_64362_104.21.69.158_443_UDP_2022-07-27 18:14:09.7052892022-07-27 14:14:09.705289192.168.43.11664362104.21.69.158443UDP12.018215183437514591665985372241628761125023908.3887677208288096153410151600837707519531250000000000000000000000474.2885746274578195880167186260223388671875000000000000000000000000224949.6520221456012222915887832641601562500000000000000000000000000000125023908.3887677208288096153410151600837707519531250000000000000000000000474.288574627457819588016718626022338867187500000000000000000000000023478.7703040000014880206435918807983398437500000000000000000000000000125023908.3887677208288096153410151600837707519531250000000000000000000000474.288574627457819588016718626022338867187500000000000000000000000065212.988211009542283136397600173950195312500000000000000000000000000014672888.00000000000000000000000000000000000000000000000000000000000000000.00000000000000000000000000000000000000000000000000000000000000003000888.00000000000000000000000000000000000000000000000000000000000000000.000000000000000000000000000000000000000000000000000000000000000011672888.00000000000000000000000000000000000000000000000000000000000000000.000000000000000000000000000000000000000000000000000000000000000099.2641116.354352296093908.38876772082880000000000138621.667194337933097.2985588958095135524.3686354421152.60169667458936121.3990596773314531.2026369972579123.89066666666666671750.57.916666666666667483352.124991371210010.2925170068038.823129251700682534980.35256846441221006950.043459147147151312970.5804830000000000000000000000000.00655658210583742479676816472533573687542229890823364257812500000.16640410606976707708071216984535567462444305419921875000000000006.9012320.012.0182150000000030.03213426470588235522862774473651370499283075332641601562500000000.36850409493750269085410309344297274947166442871093750000000000006.9012328.5e-0512.0182150000000030.00814169615912208564922902809257720946334302425384521484375000000.19116197645583432596083639509743079543113708496093750000000000007.0419710.011.870592999999998187.5729.518612.018612.0
192.168.43.116_52790_104.21.69.158_443_TCP_2022-07-27 18:14:08.5784802022-07-27 14:14:08.578480192.168.43.11652790104.21.69.158443TCP0.34346214684846305454114000346.1428571428571672186080832034349441528320312500000000000000000000561.3693534745268607366597279906272888183593750000000000000000000000315135.551020408223848789930343627929687500000000000000000000000000000014000346.1428571428571672186080832034349441528320312500000000000000000000561.369353474526860736659727990627288818359375000000000000000000000012920.138888888888686778955161571502685546875000000000000000000000000014000346.1428571428571672186080832034349441528320312500000000000000000000561.3693534745268607366597279906272888183593750000000000000000000000427336.9843750000000000000000000000000000000000000000000000000000000000304322021.71428571428571530077533680014312267303466796875000000000000000004.1991252733425907806008581246715039014816284179687500000000000000132322022.00000000000000000000000000000000000000000000000000000000000000004.4721359549995796101029554847627878189086914062500000000000000000172322021.50000000000000000000000000000000000000000000000000000000000000003.968626966596886074256644860724918544292449951171875000000000000050.833333333333336567.625346.1428571428571764240655350000000014109.27555304517888.016723829710513221.2588292154640.7614233889047423.29224193651699417.4691814523877481.33333333333333330004541.04.01795571.372083827600001454140.00252923002130011001500120018000.02642015384615384668287596525715343886986374855041503906250000000.03498305924707020148067826426085957791656255722045898437500000000.1054799.1e-050.3434620.04953079999999999982751575089423567987978458404541015625000000000.05172884125282529999001468468122766353189945220947265625000000000.1190350.0021650.247653999999999990.03399757142857142561132022251513262744992971420288085937500000000.03210960631452010327624435603866004385054111480712890625000000000.0885770.0004170.2379830000
192.168.43.116_52765_142.250.186.133_443_TCP_2022-07-27 18:14:04.3748902022-07-27 14:14:04.374890192.168.43.11652765142.250.186.133443TCP100.345666276911852048713899816587314000742.2862318840579973766580224037170410156250000000000000000000000000656.5600251477645770137314684689044952392578125000000000000000000000431071.066622033191379159688949584960937500000000000000000000000000000014000742.2862318840579973766580224037170410156250000000000000000000000000656.5600251477645770137314684689044952392578125000000000000000000000363470.203598599240649491548538208007812500000000000000000000000000000014000742.2862318840579973766580224037170410156250000000000000000000000000656.5600251477645770137314684689044952392578125000000000000000000000392090.01069393719080835580825805664062500000000000000000000000000000005592322020.26086956521739068648457759991288185119628906250000000000000000001.74996624326070504551466910925228148698806762695312500000000000001820202020.00000000000000000000000000000000000000000000000000000000000000000.00000000000000000000000000000000000000000000000000000000000000003772322020.38918918918918876670431927777826786041259765625000000000000000002.1257474086279701808166464616078883409500122070312500000000000000428.54945054945057896.6108108108108742.286231884058655272174000000002041.652700775338388.636615357159541653.01608541817862.7504924826549061.84362720757665820.90686527507824812.0329670329670337021.6666666666677.048624.5922943928831408.825.2258988.2498453927321065210.43321751570441260.6063750114000276000270009100087000185000.36489333090909092938147750828647986054420471191406250000000000003.436409533900813162432541503221727907657623291015625000000000000045.0103190.0100.3456661.11464535555555555035311954270582646131515502929687500000000000005.943249669104329058200164581649005413055419921875000000000000000045.0468320.0100.318081999999990.54504810869565223363508721377002075314521789550781250000000000004.192624593190283910359994479222223162651062011718750000000000000045.0309520.0100.2888520000000218.237.07799.67799.6
192.168.43.116_54924_142.250.185.106_443_UDP_2022-07-27 18:14:08.1274562022-07-27 14:14:08.127456192.168.43.11654924142.250.185.106443UDP0.2914931899637624403936125025354.2222222222222285381576512008905410766601562500000000000000000000469.3852428153653590925387106835842132568359375000000000000000000000220322.5061728395230602473020553588867187500000000000000000000000000000125025354.2222222222222285381576512008905410766601562500000000000000000000469.3852428153653590925387106835842132568359375000000000000000000000184884.3209876543260179460048675537109375000000000000000000000000000000125025354.2222222222222285381576512008905410766601562500000000000000000000469.3852428153653590925387106835842132568359375000000000000000000000241945.7777777777810115367174148559570312500000000000000000000000000000144888.00000000000000000000000000000000000000000000000000000000000000000.000000000000000000000000000000000000000000000000000000000000000072888.00000000000000000000000000000000000000000000000000000000000000000.000000000000000000000000000000000000000000000000000000000000000072888.00000000000000000000000000000000000000000000000000000000000000000.0000000000000000000000000000000000000000000000000000000000000000271.1111111111111437.3333333333333354.22222222222223000000000021873.5955923469858370.69843872751713502.8971536194761.751054056186630.875527028093330.87552702809331.0000000000000000000000000000000000000000.01714664705882353035182319445084431208670139312744140625000000000.02633126819389757303224719464651570888236165046691894531250000000.1000560.0001650.291493000000000060.02884574999999999986299847876125568291172385215759277343750000000.03614295245877818113511281694627541583031415939331054687500000000.1016860.0001650.2307660.02216675000000000242406095196656679036095738410949707031250000000.02386174132240772086110958127846970455721020698547363281250000000.0658750.0002850.177334000000000020000
192.168.43.116_52794_151.101.114.133_443_TCP_2022-07-27 18:14:11.1911572022-07-27 14:14:11.191157192.168.43.11652794151.101.114.133443TCP91.00038536152184252223620214000234.0277777777777714618423487991094589233398437500000000000000000000394.9627609809921864325588103383779525756835937500000000000000000000155995.582561728369910269975662231445312500000000000000000000000000000014000234.0277777777777714618423487991094589233398437500000000000000000000394.962760980992186432558810338377952575683593750000000000000000000042528.159999999996216502040624618530273437500000000000000000000000000014000234.0277777777777714618423487991094589233398437500000000000000000000394.9627609809921864325588103383779525756835937500000000000000000000228023.6507936507987324148416519165039062500000000000000000000000000000768322021.33333333333333214909544039983302354812622070312500000000000000003.7712361663282534252061850565951317548751831054687500000000000000312322020.80000000000000071054273576010018587112426757812500000000000000002.9933259094191528859596473921556025743484497070312500000000000000456322021.71428571428571530077533680014312267303466796875000000000000000004.1991252733425907806008581246715039014816284179687500000000000000148.2295.3333333333333234.0277777777777764240655350000000092.5820258892311324.42846807736033268.15355781187080.39560272190057220.230768254442000440.164834467458571751.40004796.04.04715830.8751229100001479640.00101701600235000800114000800121002.600010999999999850018639335758052766323089599609375000000000000010.439977333392109315468587737996131181716918945312500000000000000045.0083870.091.0003856.498742642857142648438184551196172833442687988281250000000000000015.747569155683976305226678960025310516357421875000000000000000000045.1380920.00021590.982396999999994.545000250000000185934823093703016638755798339843750000000000000013.512053688337649859363409632351249456405639648437500000000000000045.1353490.090.9000057.510.51111.51111.5
192.168.43.116_52834_80.66.179.18_443_TCP_2022-07-27 18:15:26.5411562022-07-27 14:15:26.541156192.168.43.1165283480.66.179.18443TCP2.82326927544972257313493735643131373140001138.3213507625273450685199350118637084960937500000000000000000000000542.6697782601822837023064494132995605468750000000000000000000000000294490.4882369554252363741397857666015625000000000000000000000000000000140001138.3213507625273450685199350118637084960937500000000000000000000000542.66977826018228370230644941329956054687500000000000000000000000001736.3027420053517744236160069704055786132812500000000000000000000000140001138.3213507625273450685199350118637084960937500000000000000000000000542.669778260182283702306449413299560546875000000000000000000000000015162.421738777933569508604705333709716796875000000000000000000000000055832402020.27305737109658778649645682889968156814575195312500000000000000002.075084834948938805609941482543945312500000000000000000000000000010680402021.48893360160965926297649275511503219604492187500000000000000000004.665509634597364119201756693655624985694885253906250000000000000045152322020.00531679220203784552722936496138572692871093750000000000000000000.25253363765078423730514600720198359340429306030273437500000000007.171026156941651387.40496233938851138.32135076252736424029200000000001110392.59808399411262.36642700359061109130.2316569907975.4649663209564799.4278972354389176.037069085517544.541247484909457582.04.032786.88524590164195188.4375140.06252555729.8648243896158240.01775116312301522411.22196604130022753000260014960003870012257000.00102552451870686531752718817500635850592516362667083740234375000.00935681054159455231900022198487931746058166027069091796875000000.3053830.02.8232689999999970.00569207459677419352200855584555938548874109983444213867187500000.02165139989682928617353852018823090475052595138549804687500000000.3053836.5e-052.8232689999999990.00121589007092198580183006750843333065859042108058929443359375000.01198274141299548491146342854563044966198503971099853515625000000.3577110.02.74304799999999640000
192.168.43.116_52838_152.199.21.118_443_TCP_2022-07-27 18:15:54.1710152022-07-27 14:15:54.171015192.168.43.11652838152.199.21.118443TCP4.65500916862811405193520936711931538140001147.8107947805456205969676375389099121093750000000000000000000000000531.9658190592804203333798795938491821289062500000000000000000000000282987.6326474110246635973453521728515625000000000000000000000000000000140001147.8107947805456205969676375389099121093750000000000000000000000000531.96581905928042033337987959384918212890625000000000000000000000004004.6364661035190692928154021501541137695312500000000000000000000000140001147.8107947805456205969676375389099121093750000000000000000000000000531.965819059280420333379879593849182128906250000000000000000000000029748.193713858738192357122898101806640625000000000000000000000000000034344322020.37010676156583599549776408821344375610351562500000000000000000002.07468121017019990759422398696187883615493774414062500000000000006232322022.17793594306049698161587002687156200408935546875000000000000000004.625129873273955638524057576432824134826660156250000000000000000028112322020.00854092526690308773140714038163423538208007812500000000000000000.320028367177711492441716245593852363526821136474609375000000000013.0640569395017781374.76014234875451147.8107947805456642406553500000000415726.15649078233788.6128684176551414937.54362236464362.19049200549347301.8254100045778660.365082000915585.0713.05.524195.737749423104275433.28571428574198.142857142857142019396.55849603621426110.0589367192803313870.95475704180021685000240012800003940011405000.00276261661721068259300881919671155628748238086700439453125000000.04543142674162266303472534900720347650349140167236328125000000001.6202740.04.6550089999999950.01662503214285714417397521458497067214921116828918457031250000000.11086401789503710912931921939161838963627815246582031250000000001.6202745.3e-054.6550089999999950.00324471082621082635918940972885593510000035166740417480468750000.05131238627027310400530168976729328278452157974243164062500000001.6410340.04.5555739999999965281.01405.03671.03671.0
192.168.43.116_52775_142.250.184.229_443_TCP_2022-07-27 18:14:06.0059342022-07-27 14:14:06.005934192.168.43.11652775142.250.184.229443TCP14.78357611561936412973017.545454545454546746441337745636701583862304687500000000000000000025.5427044783068275535242719342932105064392089843750000000000000000652.429752066115725028794258832931518554687500000000000000000000000073017.545454545454546746441337745636701583862304687500000000000000000025.5427044783068275535242719342932105064392089843750000000000000000255.759999999999990905052982270717620849609375000000000000000000000073017.545454545454546746441337745636701583862304687500000000000000000025.5427044783068275535242719342932105064392089843750000000000000000948.5833333333333712289459072053432464599609375000000000000000000000244322022.18181818181818343305167218204587697982788085937500000000000000004.6283352950392204760987624467816203832626342773437500000000000000100202020.00000000000000000000000000000000000000000000000000000000000000000.0000000000000000000000000000000000000000000000000000000000000000144322024.00000000000000000000000000000000000000000000000000000000000000005.656854249492380581898487434955313801765441894531250000000000000012.821.517.5454545454545475083740000000013.0550280933381754.3291284869100698.7258996064281060.7440689586876680.405855795647818930.33821316303984911.20000000000000014000110112000501020006001.47835760000000004943387921230169013142585754394531250000000000004.362392703804786719956609886139631271362304687500000000000000000014.5650125.4e-0514.7835759999999963.69589400000000001256239556823857128620147705078125000000000000006.344824318117523453963713109260424971580505371093750000000000000014.6852355.4e-0514.7835762.93265500000000001179500941361766308546066284179687500000000000005.816243519102170189682965428801253437995910644531250000000000000014.5650120.00033114.6632749999999995.06.064.064.0
192.168.43.116_52786_172.67.75.39_443_TCP_2022-07-27 18:15:40.4901102022-07-27 14:15:40.490110192.168.43.11652786172.67.75.39443TCP0.108553312000000.00000000000000000000000000000000000000000000000000000000000000000.00000000000000000000000000000000000000000000000000000000000000000.0000000000000000000000000000000000000000000000000000000000000000000.00000000000000000000000000000000000000000000000000000000000000000.00000000000000000000000000000000000000000000000000000000000000000.0000000000000000000000000000000000000000000000000000000000000000000.00000000000000000000000000000000000000000000000000000000000000000.00000000000000000000000000000000000000000000000000000000000000000.000000000000000000000000000000000000000000000000000000000000000060202020.00000000000000000000000000000000000000000000000000000000000000000.000000000000000000000000000000000000000000000000000000000000000020202020.00000000000000000000000000000000000000000000000000000000000000000.000000000000000000000000000000000000000000000000000000000000000040202020.00000000000000000000000000000000000000000000000000000000000000000.00000000000000000000000000000000000000000000000000000000000000000.00.00.0102095000000000.00.00.027.63626984053872318.4241798936924829.2120899468462412.0000000000000002000030010000100100002000.05427649999999999835820219118431850802153348922729492187500000000.00516849999999999948352424894437717739492654800415039062500000000.0594450.0491080.1085531658945740.49010992050170898437500000000000000000000000000000000000000000000.00000000000000000000000000000000000000000000000000000000000000001658945740.490111658945740.490111658945740.490110.04910799999999999887467794223994133062660694122314453125000000000.00000000000000000000000000000000000000000000000000000000000000000.0491080.0491080.0491080000

Copyright (c) 2023

For citation in your works and also understanding NTLFlowLyzer completely, you can find below published papers:

  • “Toward Generating a New Cloud-based Distributed Denial of Service (DDoS) Dataset and Intrusion Traffic Characterization”, MohammadMoein Shafi, Arash Habibi Lashkari, Vicente Rodriguez, and Ron Nevo, Information, Vol 15(3), 131, (2024)

Contributing

Any contribution is welcome in form of pull requests.

Project Team members

Acknowledgment

This project has been made possible through funding from the Natural Sciences and Engineering Research Council of Canada — NSERC (#RGPIN-2020-04701) and Canada Research Chair (Tier II) - (#CRC-2021-00340) to Arash Habibi Lashkari.

About

No description, website, or topics provided.

Resources

Readme
License
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages