Skip to content
BranchesTags

Repository files navigation

netml

netml is a network anomaly detection tool & library written in Python.

The library contains two primary submodules:

  • pparser: pcap parser
    Parse pcaps to produce flow features using Scapy.

  • ndm: novelty detection modeling
    Detect novelties / anomalies, via different models, such as OCSVM.

The tool's command-line interface is documented by its built-in help flags such as -h and --help:

netml --help

Installation

The netml library is available on PyPI:

pip install netml

Or, from a repository clone:

pip install .

CLI

The CLI tool is available as a distribution "extra":

pip install netml[cli]

Or:

pip install .[cli]

Tab-completion

Shell tab-completion is provided by argcomplete (through argcmdr). Completion code appropriate to your shell may be generated by register-python-argcomplete, e.g.:

register-python-argcomplete --shell=bash netml

The results of the above should be evaluated, e.g.:

eval "$(register-python-argcomplete --shell=bash netml)"

Or, to ensure the above is evaluated for every session, e.g.:

register-python-argcomplete --shell=bash netml > ~/.bash_completion

For more information, refer to argcmdr: Shell completion.

Use

Classification of network traffic for outlier detection

Having trained a model to your network traffic, the identification of anomalous traffic is as simple as providing a packet capture (PCAP) file to the netml classify command of the CLI:

netml classify --model=model.dat < unclassified.pcap

Using the Python library, the same might be accomplished, e.g.:

fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportload_datapcap=PCAP(
'unclassified.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# extract features from each flow given feat_typepcap.flow2features('IAT', fft=False, header=False)
(model, train_history) =load_data('model.dat')
model.predict(pcap.features)

Training a network traffic model

A model may be trained for outlier detection as simply as providing a PCAP file to the netml learn command:

netml learn --pcap=traffic.pcap \
--output=model.dat

(Note that for clarity and consistency with the classify command, the flags --output and --model are synonymous to the learn command.)

netml learn supports a great many additional options, documented by netml learn --help, --help-algorithm and --help-param, including:

  • --algorithm: selection of model-training algorithms, such as One-Class Support Vector Machine (OCSVM), Kernel Density Estimation (KDE), Isolation Forest (IF) and Autoencoder (AE)
  • --param: customization of model hyperparameters via YAML/JSON
  • --label, --pcap-normal & --pcap-abnormal: optional labeling of traffic to enable post-training testing of the model

In the below examples, an OCSVM model is trained by demo traffic included in the library, and tested by labels in a CSV file, (both provided by the University of New Brunswick's Intrusion Detection Systems dataset).

All of the below may be wrapped up into a single command via the CLI:

netml learn --pcap=data/demo.pcap \
--label=data/demo.csv \
--output=out/OCSVM-results.dat

PCAP to features

To only extract features via the CLI:

netml learn extract \
--pcap=data/demo.pcap \
--label=data/demo.csv \
--feature=out/IAT-features.dat

Or in Python:

fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportdump_datapcap=PCAP(
'data/demo.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# label each flow (optional)pcap.label_flows(label_file='data/demo.csv')
# extract features from each flow via IATpcap.flow2features('IAT', fft=False, header=False)
# dump data to diskdump_data((pcap.features, pcap.labels), out_file='out/IAT-features.dat')
# statsprint(pcap.features.shape, pcap.pcap2flows.tot_time, pcap.flow2features.tot_time)

Features to model

To train from already-extracted features via the CLI:

netml learn train \
--feature=out/IAT-features.dat \
--output=out/OCSVM-results.dat

Or in Python:

fromsklearn.model_selectionimporttrain_test_splitfromnetml.ndm.modelimportMODELfromnetml.ndm.ocsvmimportOCSVMfromnetml.utils.toolimportdump_data, load_dataRANDOM_STATE=42# load data
(features, labels) =load_data('out/IAT-features.dat')
# split train and test sets
(
features_train,
features_test,
labels_train,
labels_test,
) =train_test_split(features, labels, test_size=0.33, random_state=RANDOM_STATE)
# create detection modelocsvm=OCSVM(kernel='rbf', nu=0.5, random_state=RANDOM_STATE)
ocsvm.name='OCSVM'ndm=MODEL(ocsvm, score_metric='auc', verbose=10, random_state=RANDOM_STATE)
# train the model from the train setndm.train(features_train)
# evaluate the trained modelndm.test(features_test, labels_test)
# dump data to diskdump_data((ocsvm, ndm.history), out_file='out/OCSVM-results.dat')
# statsprint(ndm.train.tot_time, ndm.test.tot_time, ndm.score)

For more examples, see the examples/ directory in the source repository.

Architecture

  • examples/
    example code and datasets
  • src/netml/ndm/
    detection models (such as OCSVM)
  • src/netml/pparser/
    pcap processing (feature extraction)
  • src/netml/utils/
    common functions (such as load_data and dump_data)
  • tests/
    test cases
  • LICENSE.txt
  • manage.py
    library development & management module
  • README.md
  • setup.cfg
  • setup.py
  • tox.ini

To Do

Further work includes:

  • Evaluate pparser performance on different pcaps
  • Add test cases
  • Add examples
  • Add (generated) docs

We welcome any comments to make this tool more robust and easier to use!

Development

Development dependencies may be installed via the dev extras (below assuming a source checkout):

pip install --editable .[dev]

(Note: the installation flag --editable is also used above to instruct pip to place the source checkout directory itself onto the Python path, to ensure that any changes to the source are reflected in Python imports.)

Development tasks are then managed via argcmdr sub-commands of manage …, (as defined by the repository module manage.py), e.g.:

manage version patch -m "initial release of netml" \
--build \
--release

Thanks

netml is based on the initial work of the "Outlier Detection" library odet 🙌

About

Network anomaly detection via machine learning

Resources

Readme
License
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GitHub - shinan6/netml: Network anomaly detection via machine learning · GitHub
Skip to content
BranchesTags
Repository files navigation
netml
netml is a network anomaly detection tool & library written in Python.
The library contains two primary submodules:
  • pparser: pcap parser
    
Parse pcaps to produce flow features using Scapy.
  • ndm: novelty detection modeling
    
Detect novelties / anomalies, via different models, such as OCSVM.
The tool's command-line interface is documented by its built-in help flags such as -h and --help:
netml --help

Installation
The netml library is available on PyPI:
pip install netml

Or, from a repository clone:
pip install .

CLI
The CLI tool is available as a distribution "extra":
pip install netml[cli]

Or:
pip install .[cli]

Tab-completion
Shell tab-completion is provided by argcomplete (through argcmdr). Completion code appropriate to your shell may be generated by register-python-argcomplete, e.g.:
register-python-argcomplete --shell=bash netml

The results of the above should be evaluated, e.g.:
eval "$(register-python-argcomplete --shell=bash netml)"

Or, to ensure the above is evaluated for every session, e.g.:
register-python-argcomplete --shell=bash netml > ~/.bash_completion

For more information, refer to argcmdr: Shell completion.
Use
Classification of network traffic for outlier detection
Having trained a model to your network traffic, the identification of anomalous traffic is as simple as providing a packet capture (PCAP) file to the netml classify command of the CLI:
netml classify --model=model.dat < unclassified.pcap

Using the Python library, the same might be accomplished, e.g.:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportload_datapcap=PCAP(
'unclassified.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# extract features from each flow given feat_typepcap.flow2features('IAT', fft=False, header=False)
(model, train_history) =load_data('model.dat')
model.predict(pcap.features)
Training a network traffic model
A model may be trained for outlier detection as simply as providing a PCAP file to the netml learn command:
netml learn --pcap=traffic.pcap \
--output=model.dat

(Note that for clarity and consistency with the classify command, the flags --output and --model are synonymous to the learn command.)
netml learn supports a great many additional options, documented by netml learn --help, --help-algorithm and --help-param, including:
  • --algorithm: selection of model-training algorithms, such as One-Class Support Vector Machine (OCSVM), Kernel Density Estimation (KDE), Isolation Forest (IF) and Autoencoder (AE)
  • --param: customization of model hyperparameters via YAML/JSON
  • --label, --pcap-normal & --pcap-abnormal: optional labeling of traffic to enable post-training testing of the model
In the below examples, an OCSVM model is trained by demo traffic included in the library, and tested by labels in a CSV file, (both provided by the University of New Brunswick's Intrusion Detection Systems dataset).
All of the below may be wrapped up into a single command via the CLI:
netml learn --pcap=data/demo.pcap \
--label=data/demo.csv \
--output=out/OCSVM-results.dat

PCAP to features
To only extract features via the CLI:
netml learn extract \
--pcap=data/demo.pcap \
--label=data/demo.csv \
--feature=out/IAT-features.dat

Or in Python:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportdump_datapcap=PCAP(
'data/demo.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# label each flow (optional)pcap.label_flows(label_file='data/demo.csv')
# extract features from each flow via IATpcap.flow2features('IAT', fft=False, header=False)
# dump data to diskdump_data((pcap.features, pcap.labels), out_file='out/IAT-features.dat')
# statsprint(pcap.features.shape, pcap.pcap2flows.tot_time, pcap.flow2features.tot_time)
Features to model
To train from already-extracted features via the CLI:
netml learn train \
--feature=out/IAT-features.dat \
--output=out/OCSVM-results.dat

Or in Python:
fromsklearn.model_selectionimporttrain_test_splitfromnetml.ndm.modelimportMODELfromnetml.ndm.ocsvmimportOCSVMfromnetml.utils.toolimportdump_data, load_dataRANDOM_STATE=42# load data
(features, labels) =load_data('out/IAT-features.dat')
# split train and test sets
(
features_train,
features_test,
labels_train,
labels_test,
) =train_test_split(features, labels, test_size=0.33, random_state=RANDOM_STATE)
# create detection modelocsvm=OCSVM(kernel='rbf', nu=0.5, random_state=RANDOM_STATE)
ocsvm.name='OCSVM'ndm=MODEL(ocsvm, score_metric='auc', verbose=10, random_state=RANDOM_STATE)
# train the model from the train setndm.train(features_train)
# evaluate the trained modelndm.test(features_test, labels_test)
# dump data to diskdump_data((ocsvm, ndm.history), out_file='out/OCSVM-results.dat')
# statsprint(ndm.train.tot_time, ndm.test.tot_time, ndm.score)
For more examples, see the examples/ directory in the source repository.
Architecture
  • examples/
    
example code and datasets
  • src/netml/ndm/
    
detection models (such as OCSVM)
  • src/netml/pparser/
    
pcap processing (feature extraction)
  • src/netml/utils/
    
common functions (such as load_data and dump_data)
  • tests/
    
test cases
  • LICENSE.txt
  • manage.py
    
library development & management module
  • README.md
  • setup.cfg
  • setup.py
  • tox.ini
To Do
Further work includes:
  • Evaluate pparser performance on different pcaps
  • Add test cases
  • Add examples
  • Add (generated) docs
We welcome any comments to make this tool more robust and easier to use!
Development
Development dependencies may be installed via the dev extras (below assuming a source checkout):
pip install --editable .[dev]

(Note: the installation flag --editable is also used above to instruct pip to place the source checkout directory itself onto the Python path, to ensure that any changes to the source are reflected in Python imports.)
Development tasks are then managed via argcmdr sub-commands of manage …, (as defined by the repository module manage.py), e.g.:
manage version patch -m "initial release of netml" \
--build \
--release

Thanks
netml is based on the initial work of the "Outlier Detection" library odet 🙌
About
Network anomaly detection via machine learning
Resources
Readme
License
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - shinan6/netml: Network anomaly detection via machine learning · GitHub
Skip to content
BranchesTags
Repository files navigation
netml
netml is a network anomaly detection tool & library written in Python.
The library contains two primary submodules:
  • pparser: pcap parser
    
Parse pcaps to produce flow features using Scapy.
  • ndm: novelty detection modeling
    
Detect novelties / anomalies, via different models, such as OCSVM.
The tool's command-line interface is documented by its built-in help flags such as -h and --help:
netml --help

Installation
The netml library is available on PyPI:
pip install netml

Or, from a repository clone:
pip install .

CLI
The CLI tool is available as a distribution "extra":
pip install netml[cli]

Or:
pip install .[cli]

Tab-completion
Shell tab-completion is provided by argcomplete (through argcmdr). Completion code appropriate to your shell may be generated by register-python-argcomplete, e.g.:
register-python-argcomplete --shell=bash netml

The results of the above should be evaluated, e.g.:
eval "$(register-python-argcomplete --shell=bash netml)"

Or, to ensure the above is evaluated for every session, e.g.:
register-python-argcomplete --shell=bash netml > ~/.bash_completion

For more information, refer to argcmdr: Shell completion.
Use
Classification of network traffic for outlier detection
Having trained a model to your network traffic, the identification of anomalous traffic is as simple as providing a packet capture (PCAP) file to the netml classify command of the CLI:
netml classify --model=model.dat < unclassified.pcap

Using the Python library, the same might be accomplished, e.g.:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportload_datapcap=PCAP(
'unclassified.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# extract features from each flow given feat_typepcap.flow2features('IAT', fft=False, header=False)
(model, train_history) =load_data('model.dat')
model.predict(pcap.features)
Training a network traffic model
A model may be trained for outlier detection as simply as providing a PCAP file to the netml learn command:
netml learn --pcap=traffic.pcap \
--output=model.dat

(Note that for clarity and consistency with the classify command, the flags --output and --model are synonymous to the learn command.)
netml learn supports a great many additional options, documented by netml learn --help, --help-algorithm and --help-param, including:
  • --algorithm: selection of model-training algorithms, such as One-Class Support Vector Machine (OCSVM), Kernel Density Estimation (KDE), Isolation Forest (IF) and Autoencoder (AE)
  • --param: customization of model hyperparameters via YAML/JSON
  • --label, --pcap-normal & --pcap-abnormal: optional labeling of traffic to enable post-training testing of the model
In the below examples, an OCSVM model is trained by demo traffic included in the library, and tested by labels in a CSV file, (both provided by the University of New Brunswick's Intrusion Detection Systems dataset).
All of the below may be wrapped up into a single command via the CLI:
netml learn --pcap=data/demo.pcap \
--label=data/demo.csv \
--output=out/OCSVM-results.dat

PCAP to features
To only extract features via the CLI:
netml learn extract \
--pcap=data/demo.pcap \
--label=data/demo.csv \
--feature=out/IAT-features.dat

Or in Python:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportdump_datapcap=PCAP(
'data/demo.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# label each flow (optional)pcap.label_flows(label_file='data/demo.csv')
# extract features from each flow via IATpcap.flow2features('IAT', fft=False, header=False)
# dump data to diskdump_data((pcap.features, pcap.labels), out_file='out/IAT-features.dat')
# statsprint(pcap.features.shape, pcap.pcap2flows.tot_time, pcap.flow2features.tot_time)
Features to model
To train from already-extracted features via the CLI:
netml learn train \
--feature=out/IAT-features.dat \
--output=out/OCSVM-results.dat

Or in Python:
fromsklearn.model_selectionimporttrain_test_splitfromnetml.ndm.modelimportMODELfromnetml.ndm.ocsvmimportOCSVMfromnetml.utils.toolimportdump_data, load_dataRANDOM_STATE=42# load data
(features, labels) =load_data('out/IAT-features.dat')
# split train and test sets
(
features_train,
features_test,
labels_train,
labels_test,
) =train_test_split(features, labels, test_size=0.33, random_state=RANDOM_STATE)
# create detection modelocsvm=OCSVM(kernel='rbf', nu=0.5, random_state=RANDOM_STATE)
ocsvm.name='OCSVM'ndm=MODEL(ocsvm, score_metric='auc', verbose=10, random_state=RANDOM_STATE)
# train the model from the train setndm.train(features_train)
# evaluate the trained modelndm.test(features_test, labels_test)
# dump data to diskdump_data((ocsvm, ndm.history), out_file='out/OCSVM-results.dat')
# statsprint(ndm.train.tot_time, ndm.test.tot_time, ndm.score)
For more examples, see the examples/ directory in the source repository.
Architecture
  • examples/
    
example code and datasets
  • src/netml/ndm/
    
detection models (such as OCSVM)
  • src/netml/pparser/
    
pcap processing (feature extraction)
  • src/netml/utils/
    
common functions (such as load_data and dump_data)
  • tests/
    
test cases
  • LICENSE.txt
  • manage.py
    
library development & management module
  • README.md
  • setup.cfg
  • setup.py
  • tox.ini
To Do
Further work includes:
  • Evaluate pparser performance on different pcaps
  • Add test cases
  • Add examples
  • Add (generated) docs
We welcome any comments to make this tool more robust and easier to use!
Development
Development dependencies may be installed via the dev extras (below assuming a source checkout):
pip install --editable .[dev]

(Note: the installation flag --editable is also used above to instruct pip to place the source checkout directory itself onto the Python path, to ensure that any changes to the source are reflected in Python imports.)
Development tasks are then managed via argcmdr sub-commands of manage …, (as defined by the repository module manage.py), e.g.:
manage version patch -m "initial release of netml" \
--build \
--release

Thanks
netml is based on the initial work of the "Outlier Detection" library odet 🙌
About
Network anomaly detection via machine learning
Resources
Readme
License
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - shinan6/netml: Network anomaly detection via machine learning · GitHub
Skip to content
BranchesTags
Repository files navigation
netml
netml is a network anomaly detection tool & library written in Python.
The library contains two primary submodules:
  • pparser: pcap parser
    
Parse pcaps to produce flow features using Scapy.
  • ndm: novelty detection modeling
    
Detect novelties / anomalies, via different models, such as OCSVM.
The tool's command-line interface is documented by its built-in help flags such as -h and --help:
netml --help

Installation
The netml library is available on PyPI:
pip install netml

Or, from a repository clone:
pip install .

CLI
The CLI tool is available as a distribution "extra":
pip install netml[cli]

Or:
pip install .[cli]

Tab-completion
Shell tab-completion is provided by argcomplete (through argcmdr). Completion code appropriate to your shell may be generated by register-python-argcomplete, e.g.:
register-python-argcomplete --shell=bash netml

The results of the above should be evaluated, e.g.:
eval "$(register-python-argcomplete --shell=bash netml)"

Or, to ensure the above is evaluated for every session, e.g.:
register-python-argcomplete --shell=bash netml > ~/.bash_completion

For more information, refer to argcmdr: Shell completion.
Use
Classification of network traffic for outlier detection
Having trained a model to your network traffic, the identification of anomalous traffic is as simple as providing a packet capture (PCAP) file to the netml classify command of the CLI:
netml classify --model=model.dat < unclassified.pcap

Using the Python library, the same might be accomplished, e.g.:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportload_datapcap=PCAP(
'unclassified.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# extract features from each flow given feat_typepcap.flow2features('IAT', fft=False, header=False)
(model, train_history) =load_data('model.dat')
model.predict(pcap.features)
Training a network traffic model
A model may be trained for outlier detection as simply as providing a PCAP file to the netml learn command:
netml learn --pcap=traffic.pcap \
--output=model.dat

(Note that for clarity and consistency with the classify command, the flags --output and --model are synonymous to the learn command.)
netml learn supports a great many additional options, documented by netml learn --help, --help-algorithm and --help-param, including:
  • --algorithm: selection of model-training algorithms, such as One-Class Support Vector Machine (OCSVM), Kernel Density Estimation (KDE), Isolation Forest (IF) and Autoencoder (AE)
  • --param: customization of model hyperparameters via YAML/JSON
  • --label, --pcap-normal & --pcap-abnormal: optional labeling of traffic to enable post-training testing of the model
In the below examples, an OCSVM model is trained by demo traffic included in the library, and tested by labels in a CSV file, (both provided by the University of New Brunswick's Intrusion Detection Systems dataset).
All of the below may be wrapped up into a single command via the CLI:
netml learn --pcap=data/demo.pcap \
--label=data/demo.csv \
--output=out/OCSVM-results.dat

PCAP to features
To only extract features via the CLI:
netml learn extract \
--pcap=data/demo.pcap \
--label=data/demo.csv \
--feature=out/IAT-features.dat

Or in Python:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportdump_datapcap=PCAP(
'data/demo.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# label each flow (optional)pcap.label_flows(label_file='data/demo.csv')
# extract features from each flow via IATpcap.flow2features('IAT', fft=False, header=False)
# dump data to diskdump_data((pcap.features, pcap.labels), out_file='out/IAT-features.dat')
# statsprint(pcap.features.shape, pcap.pcap2flows.tot_time, pcap.flow2features.tot_time)
Features to model
To train from already-extracted features via the CLI:
netml learn train \
--feature=out/IAT-features.dat \
--output=out/OCSVM-results.dat

Or in Python:
fromsklearn.model_selectionimporttrain_test_splitfromnetml.ndm.modelimportMODELfromnetml.ndm.ocsvmimportOCSVMfromnetml.utils.toolimportdump_data, load_dataRANDOM_STATE=42# load data
(features, labels) =load_data('out/IAT-features.dat')
# split train and test sets
(
features_train,
features_test,
labels_train,
labels_test,
) =train_test_split(features, labels, test_size=0.33, random_state=RANDOM_STATE)
# create detection modelocsvm=OCSVM(kernel='rbf', nu=0.5, random_state=RANDOM_STATE)
ocsvm.name='OCSVM'ndm=MODEL(ocsvm, score_metric='auc', verbose=10, random_state=RANDOM_STATE)
# train the model from the train setndm.train(features_train)
# evaluate the trained modelndm.test(features_test, labels_test)
# dump data to diskdump_data((ocsvm, ndm.history), out_file='out/OCSVM-results.dat')
# statsprint(ndm.train.tot_time, ndm.test.tot_time, ndm.score)
For more examples, see the examples/ directory in the source repository.
Architecture
  • examples/
    
example code and datasets
  • src/netml/ndm/
    
detection models (such as OCSVM)
  • src/netml/pparser/
    
pcap processing (feature extraction)
  • src/netml/utils/
    
common functions (such as load_data and dump_data)
  • tests/
    
test cases
  • LICENSE.txt
  • manage.py
    
library development & management module
  • README.md
  • setup.cfg
  • setup.py
  • tox.ini
To Do
Further work includes:
  • Evaluate pparser performance on different pcaps
  • Add test cases
  • Add examples
  • Add (generated) docs
We welcome any comments to make this tool more robust and easier to use!
Development
Development dependencies may be installed via the dev extras (below assuming a source checkout):
pip install --editable .[dev]

(Note: the installation flag --editable is also used above to instruct pip to place the source checkout directory itself onto the Python path, to ensure that any changes to the source are reflected in Python imports.)
Development tasks are then managed via argcmdr sub-commands of manage …, (as defined by the repository module manage.py), e.g.:
manage version patch -m "initial release of netml" \
--build \
--release

Thanks
netml is based on the initial work of the "Outlier Detection" library odet 🙌
About
Network anomaly detection via machine learning
Resources
Readme
License
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
GitHub - shinan6/netml: Network anomaly detection via machine learning · GitHub
Skip to content
BranchesTags
Repository files navigation
netml
netml is a network anomaly detection tool & library written in Python.
The library contains two primary submodules:
  • pparser: pcap parser
    
Parse pcaps to produce flow features using Scapy.
  • ndm: novelty detection modeling
    
Detect novelties / anomalies, via different models, such as OCSVM.
The tool's command-line interface is documented by its built-in help flags such as -h and --help:
netml --help

Installation
The netml library is available on PyPI:
pip install netml

Or, from a repository clone:
pip install .

CLI
The CLI tool is available as a distribution "extra":
pip install netml[cli]

Or:
pip install .[cli]

Tab-completion
Shell tab-completion is provided by argcomplete (through argcmdr). Completion code appropriate to your shell may be generated by register-python-argcomplete, e.g.:
register-python-argcomplete --shell=bash netml

The results of the above should be evaluated, e.g.:
eval "$(register-python-argcomplete --shell=bash netml)"

Or, to ensure the above is evaluated for every session, e.g.:
register-python-argcomplete --shell=bash netml > ~/.bash_completion

For more information, refer to argcmdr: Shell completion.
Use
Classification of network traffic for outlier detection
Having trained a model to your network traffic, the identification of anomalous traffic is as simple as providing a packet capture (PCAP) file to the netml classify command of the CLI:
netml classify --model=model.dat < unclassified.pcap

Using the Python library, the same might be accomplished, e.g.:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportload_datapcap=PCAP(
'unclassified.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# extract features from each flow given feat_typepcap.flow2features('IAT', fft=False, header=False)
(model, train_history) =load_data('model.dat')
model.predict(pcap.features)
Training a network traffic model
A model may be trained for outlier detection as simply as providing a PCAP file to the netml learn command:
netml learn --pcap=traffic.pcap \
--output=model.dat

(Note that for clarity and consistency with the classify command, the flags --output and --model are synonymous to the learn command.)
netml learn supports a great many additional options, documented by netml learn --help, --help-algorithm and --help-param, including:
  • --algorithm: selection of model-training algorithms, such as One-Class Support Vector Machine (OCSVM), Kernel Density Estimation (KDE), Isolation Forest (IF) and Autoencoder (AE)
  • --param: customization of model hyperparameters via YAML/JSON
  • --label, --pcap-normal & --pcap-abnormal: optional labeling of traffic to enable post-training testing of the model
In the below examples, an OCSVM model is trained by demo traffic included in the library, and tested by labels in a CSV file, (both provided by the University of New Brunswick's Intrusion Detection Systems dataset).
All of the below may be wrapped up into a single command via the CLI:
netml learn --pcap=data/demo.pcap \
--label=data/demo.csv \
--output=out/OCSVM-results.dat

PCAP to features
To only extract features via the CLI:
netml learn extract \
--pcap=data/demo.pcap \
--label=data/demo.csv \
--feature=out/IAT-features.dat

Or in Python:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportdump_datapcap=PCAP(
'data/demo.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# label each flow (optional)pcap.label_flows(label_file='data/demo.csv')
# extract features from each flow via IATpcap.flow2features('IAT', fft=False, header=False)
# dump data to diskdump_data((pcap.features, pcap.labels), out_file='out/IAT-features.dat')
# statsprint(pcap.features.shape, pcap.pcap2flows.tot_time, pcap.flow2features.tot_time)
Features to model
To train from already-extracted features via the CLI:
netml learn train \
--feature=out/IAT-features.dat \
--output=out/OCSVM-results.dat

Or in Python:
fromsklearn.model_selectionimporttrain_test_splitfromnetml.ndm.modelimportMODELfromnetml.ndm.ocsvmimportOCSVMfromnetml.utils.toolimportdump_data, load_dataRANDOM_STATE=42# load data
(features, labels) =load_data('out/IAT-features.dat')
# split train and test sets
(
features_train,
features_test,
labels_train,
labels_test,
) =train_test_split(features, labels, test_size=0.33, random_state=RANDOM_STATE)
# create detection modelocsvm=OCSVM(kernel='rbf', nu=0.5, random_state=RANDOM_STATE)
ocsvm.name='OCSVM'ndm=MODEL(ocsvm, score_metric='auc', verbose=10, random_state=RANDOM_STATE)
# train the model from the train setndm.train(features_train)
# evaluate the trained modelndm.test(features_test, labels_test)
# dump data to diskdump_data((ocsvm, ndm.history), out_file='out/OCSVM-results.dat')
# statsprint(ndm.train.tot_time, ndm.test.tot_time, ndm.score)
For more examples, see the examples/ directory in the source repository.
Architecture
  • examples/
    
example code and datasets
  • src/netml/ndm/
    
detection models (such as OCSVM)
  • src/netml/pparser/
    
pcap processing (feature extraction)
  • src/netml/utils/
    
common functions (such as load_data and dump_data)
  • tests/
    
test cases
  • LICENSE.txt
  • manage.py
    
library development & management module
  • README.md
  • setup.cfg
  • setup.py
  • tox.ini
To Do
Further work includes:
  • Evaluate pparser performance on different pcaps
  • Add test cases
  • Add examples
  • Add (generated) docs
We welcome any comments to make this tool more robust and easier to use!
Development
Development dependencies may be installed via the dev extras (below assuming a source checkout):
pip install --editable .[dev]

(Note: the installation flag --editable is also used above to instruct pip to place the source checkout directory itself onto the Python path, to ensure that any changes to the source are reflected in Python imports.)
Development tasks are then managed via argcmdr sub-commands of manage …, (as defined by the repository module manage.py), e.g.:
manage version patch -m "initial release of netml" \
--build \
--release

Thanks
netml is based on the initial work of the "Outlier Detection" library odet 🙌
About
Network anomaly detection via machine learning
Resources
Readme
License
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - shinan6/netml: Network anomaly detection via machine learning · GitHub
Skip to content
BranchesTags
Repository files navigation
netml
netml is a network anomaly detection tool & library written in Python.
The library contains two primary submodules:
  • pparser: pcap parser
    
Parse pcaps to produce flow features using Scapy.
  • ndm: novelty detection modeling
    
Detect novelties / anomalies, via different models, such as OCSVM.
The tool's command-line interface is documented by its built-in help flags such as -h and --help:
netml --help

Installation
The netml library is available on PyPI:
pip install netml

Or, from a repository clone:
pip install .

CLI
The CLI tool is available as a distribution "extra":
pip install netml[cli]

Or:
pip install .[cli]

Tab-completion
Shell tab-completion is provided by argcomplete (through argcmdr). Completion code appropriate to your shell may be generated by register-python-argcomplete, e.g.:
register-python-argcomplete --shell=bash netml

The results of the above should be evaluated, e.g.:
eval "$(register-python-argcomplete --shell=bash netml)"

Or, to ensure the above is evaluated for every session, e.g.:
register-python-argcomplete --shell=bash netml > ~/.bash_completion

For more information, refer to argcmdr: Shell completion.
Use
Classification of network traffic for outlier detection
Having trained a model to your network traffic, the identification of anomalous traffic is as simple as providing a packet capture (PCAP) file to the netml classify command of the CLI:
netml classify --model=model.dat < unclassified.pcap

Using the Python library, the same might be accomplished, e.g.:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportload_datapcap=PCAP(
'unclassified.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# extract features from each flow given feat_typepcap.flow2features('IAT', fft=False, header=False)
(model, train_history) =load_data('model.dat')
model.predict(pcap.features)
Training a network traffic model
A model may be trained for outlier detection as simply as providing a PCAP file to the netml learn command:
netml learn --pcap=traffic.pcap \
--output=model.dat

(Note that for clarity and consistency with the classify command, the flags --output and --model are synonymous to the learn command.)
netml learn supports a great many additional options, documented by netml learn --help, --help-algorithm and --help-param, including:
  • --algorithm: selection of model-training algorithms, such as One-Class Support Vector Machine (OCSVM), Kernel Density Estimation (KDE), Isolation Forest (IF) and Autoencoder (AE)
  • --param: customization of model hyperparameters via YAML/JSON
  • --label, --pcap-normal & --pcap-abnormal: optional labeling of traffic to enable post-training testing of the model
In the below examples, an OCSVM model is trained by demo traffic included in the library, and tested by labels in a CSV file, (both provided by the University of New Brunswick's Intrusion Detection Systems dataset).
All of the below may be wrapped up into a single command via the CLI:
netml learn --pcap=data/demo.pcap \
--label=data/demo.csv \
--output=out/OCSVM-results.dat

PCAP to features
To only extract features via the CLI:
netml learn extract \
--pcap=data/demo.pcap \
--label=data/demo.csv \
--feature=out/IAT-features.dat

Or in Python:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportdump_datapcap=PCAP(
'data/demo.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# label each flow (optional)pcap.label_flows(label_file='data/demo.csv')
# extract features from each flow via IATpcap.flow2features('IAT', fft=False, header=False)
# dump data to diskdump_data((pcap.features, pcap.labels), out_file='out/IAT-features.dat')
# statsprint(pcap.features.shape, pcap.pcap2flows.tot_time, pcap.flow2features.tot_time)
Features to model
To train from already-extracted features via the CLI:
netml learn train \
--feature=out/IAT-features.dat \
--output=out/OCSVM-results.dat

Or in Python:
fromsklearn.model_selectionimporttrain_test_splitfromnetml.ndm.modelimportMODELfromnetml.ndm.ocsvmimportOCSVMfromnetml.utils.toolimportdump_data, load_dataRANDOM_STATE=42# load data
(features, labels) =load_data('out/IAT-features.dat')
# split train and test sets
(
features_train,
features_test,
labels_train,
labels_test,
) =train_test_split(features, labels, test_size=0.33, random_state=RANDOM_STATE)
# create detection modelocsvm=OCSVM(kernel='rbf', nu=0.5, random_state=RANDOM_STATE)
ocsvm.name='OCSVM'ndm=MODEL(ocsvm, score_metric='auc', verbose=10, random_state=RANDOM_STATE)
# train the model from the train setndm.train(features_train)
# evaluate the trained modelndm.test(features_test, labels_test)
# dump data to diskdump_data((ocsvm, ndm.history), out_file='out/OCSVM-results.dat')
# statsprint(ndm.train.tot_time, ndm.test.tot_time, ndm.score)
For more examples, see the examples/ directory in the source repository.
Architecture
  • examples/
    
example code and datasets
  • src/netml/ndm/
    
detection models (such as OCSVM)
  • src/netml/pparser/
    
pcap processing (feature extraction)
  • src/netml/utils/
    
common functions (such as load_data and dump_data)
  • tests/
    
test cases
  • LICENSE.txt
  • manage.py
    
library development & management module
  • README.md
  • setup.cfg
  • setup.py
  • tox.ini
To Do
Further work includes:
  • Evaluate pparser performance on different pcaps
  • Add test cases
  • Add examples
  • Add (generated) docs
We welcome any comments to make this tool more robust and easier to use!
Development
Development dependencies may be installed via the dev extras (below assuming a source checkout):
pip install --editable .[dev]

(Note: the installation flag --editable is also used above to instruct pip to place the source checkout directory itself onto the Python path, to ensure that any changes to the source are reflected in Python imports.)
Development tasks are then managed via argcmdr sub-commands of manage …, (as defined by the repository module manage.py), e.g.:
manage version patch -m "initial release of netml" \
--build \
--release

Thanks
netml is based on the initial work of the "Outlier Detection" library odet 🙌
About
Network anomaly detection via machine learning
Resources
Readme
License
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - shinan6/netml: Network anomaly detection via machine learning · GitHub
Skip to content
BranchesTags
Repository files navigation
netml
netml is a network anomaly detection tool & library written in Python.
The library contains two primary submodules:
  • pparser: pcap parser
    
Parse pcaps to produce flow features using Scapy.
  • ndm: novelty detection modeling
    
Detect novelties / anomalies, via different models, such as OCSVM.
The tool's command-line interface is documented by its built-in help flags such as -h and --help:
netml --help

Installation
The netml library is available on PyPI:
pip install netml

Or, from a repository clone:
pip install .

CLI
The CLI tool is available as a distribution "extra":
pip install netml[cli]

Or:
pip install .[cli]

Tab-completion
Shell tab-completion is provided by argcomplete (through argcmdr). Completion code appropriate to your shell may be generated by register-python-argcomplete, e.g.:
register-python-argcomplete --shell=bash netml

The results of the above should be evaluated, e.g.:
eval "$(register-python-argcomplete --shell=bash netml)"

Or, to ensure the above is evaluated for every session, e.g.:
register-python-argcomplete --shell=bash netml > ~/.bash_completion

For more information, refer to argcmdr: Shell completion.
Use
Classification of network traffic for outlier detection
Having trained a model to your network traffic, the identification of anomalous traffic is as simple as providing a packet capture (PCAP) file to the netml classify command of the CLI:
netml classify --model=model.dat < unclassified.pcap

Using the Python library, the same might be accomplished, e.g.:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportload_datapcap=PCAP(
'unclassified.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# extract features from each flow given feat_typepcap.flow2features('IAT', fft=False, header=False)
(model, train_history) =load_data('model.dat')
model.predict(pcap.features)
Training a network traffic model
A model may be trained for outlier detection as simply as providing a PCAP file to the netml learn command:
netml learn --pcap=traffic.pcap \
--output=model.dat

(Note that for clarity and consistency with the classify command, the flags --output and --model are synonymous to the learn command.)
netml learn supports a great many additional options, documented by netml learn --help, --help-algorithm and --help-param, including:
  • --algorithm: selection of model-training algorithms, such as One-Class Support Vector Machine (OCSVM), Kernel Density Estimation (KDE), Isolation Forest (IF) and Autoencoder (AE)
  • --param: customization of model hyperparameters via YAML/JSON
  • --label, --pcap-normal & --pcap-abnormal: optional labeling of traffic to enable post-training testing of the model
In the below examples, an OCSVM model is trained by demo traffic included in the library, and tested by labels in a CSV file, (both provided by the University of New Brunswick's Intrusion Detection Systems dataset).
All of the below may be wrapped up into a single command via the CLI:
netml learn --pcap=data/demo.pcap \
--label=data/demo.csv \
--output=out/OCSVM-results.dat

PCAP to features
To only extract features via the CLI:
netml learn extract \
--pcap=data/demo.pcap \
--label=data/demo.csv \
--feature=out/IAT-features.dat

Or in Python:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportdump_datapcap=PCAP(
'data/demo.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# label each flow (optional)pcap.label_flows(label_file='data/demo.csv')
# extract features from each flow via IATpcap.flow2features('IAT', fft=False, header=False)
# dump data to diskdump_data((pcap.features, pcap.labels), out_file='out/IAT-features.dat')
# statsprint(pcap.features.shape, pcap.pcap2flows.tot_time, pcap.flow2features.tot_time)
Features to model
To train from already-extracted features via the CLI:
netml learn train \
--feature=out/IAT-features.dat \
--output=out/OCSVM-results.dat

Or in Python:
fromsklearn.model_selectionimporttrain_test_splitfromnetml.ndm.modelimportMODELfromnetml.ndm.ocsvmimportOCSVMfromnetml.utils.toolimportdump_data, load_dataRANDOM_STATE=42# load data
(features, labels) =load_data('out/IAT-features.dat')
# split train and test sets
(
features_train,
features_test,
labels_train,
labels_test,
) =train_test_split(features, labels, test_size=0.33, random_state=RANDOM_STATE)
# create detection modelocsvm=OCSVM(kernel='rbf', nu=0.5, random_state=RANDOM_STATE)
ocsvm.name='OCSVM'ndm=MODEL(ocsvm, score_metric='auc', verbose=10, random_state=RANDOM_STATE)
# train the model from the train setndm.train(features_train)
# evaluate the trained modelndm.test(features_test, labels_test)
# dump data to diskdump_data((ocsvm, ndm.history), out_file='out/OCSVM-results.dat')
# statsprint(ndm.train.tot_time, ndm.test.tot_time, ndm.score)
For more examples, see the examples/ directory in the source repository.
Architecture
  • examples/
    
example code and datasets
  • src/netml/ndm/
    
detection models (such as OCSVM)
  • src/netml/pparser/
    
pcap processing (feature extraction)
  • src/netml/utils/
    
common functions (such as load_data and dump_data)
  • tests/
    
test cases
  • LICENSE.txt
  • manage.py
    
library development & management module
  • README.md
  • setup.cfg
  • setup.py
  • tox.ini
To Do
Further work includes:
  • Evaluate pparser performance on different pcaps
  • Add test cases
  • Add examples
  • Add (generated) docs
We welcome any comments to make this tool more robust and easier to use!
Development
Development dependencies may be installed via the dev extras (below assuming a source checkout):
pip install --editable .[dev]

(Note: the installation flag --editable is also used above to instruct pip to place the source checkout directory itself onto the Python path, to ensure that any changes to the source are reflected in Python imports.)
Development tasks are then managed via argcmdr sub-commands of manage …, (as defined by the repository module manage.py), e.g.:
manage version patch -m "initial release of netml" \
--build \
--release

Thanks
netml is based on the initial work of the "Outlier Detection" library odet 🙌
About
Network anomaly detection via machine learning
Resources
Readme
License
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
GitHub - shinan6/netml: Network anomaly detection via machine learning · GitHub
Skip to content
BranchesTags
Repository files navigation
netml
netml is a network anomaly detection tool & library written in Python.
The library contains two primary submodules:
  • pparser: pcap parser
    
Parse pcaps to produce flow features using Scapy.
  • ndm: novelty detection modeling
    
Detect novelties / anomalies, via different models, such as OCSVM.
The tool's command-line interface is documented by its built-in help flags such as -h and --help:
netml --help

Installation
The netml library is available on PyPI:
pip install netml

Or, from a repository clone:
pip install .

CLI
The CLI tool is available as a distribution "extra":
pip install netml[cli]

Or:
pip install .[cli]

Tab-completion
Shell tab-completion is provided by argcomplete (through argcmdr). Completion code appropriate to your shell may be generated by register-python-argcomplete, e.g.:
register-python-argcomplete --shell=bash netml

The results of the above should be evaluated, e.g.:
eval "$(register-python-argcomplete --shell=bash netml)"

Or, to ensure the above is evaluated for every session, e.g.:
register-python-argcomplete --shell=bash netml > ~/.bash_completion

For more information, refer to argcmdr: Shell completion.
Use
Classification of network traffic for outlier detection
Having trained a model to your network traffic, the identification of anomalous traffic is as simple as providing a packet capture (PCAP) file to the netml classify command of the CLI:
netml classify --model=model.dat < unclassified.pcap

Using the Python library, the same might be accomplished, e.g.:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportload_datapcap=PCAP(
'unclassified.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# extract features from each flow given feat_typepcap.flow2features('IAT', fft=False, header=False)
(model, train_history) =load_data('model.dat')
model.predict(pcap.features)
Training a network traffic model
A model may be trained for outlier detection as simply as providing a PCAP file to the netml learn command:
netml learn --pcap=traffic.pcap \
--output=model.dat

(Note that for clarity and consistency with the classify command, the flags --output and --model are synonymous to the learn command.)
netml learn supports a great many additional options, documented by netml learn --help, --help-algorithm and --help-param, including:
  • --algorithm: selection of model-training algorithms, such as One-Class Support Vector Machine (OCSVM), Kernel Density Estimation (KDE), Isolation Forest (IF) and Autoencoder (AE)
  • --param: customization of model hyperparameters via YAML/JSON
  • --label, --pcap-normal & --pcap-abnormal: optional labeling of traffic to enable post-training testing of the model
In the below examples, an OCSVM model is trained by demo traffic included in the library, and tested by labels in a CSV file, (both provided by the University of New Brunswick's Intrusion Detection Systems dataset).
All of the below may be wrapped up into a single command via the CLI:
netml learn --pcap=data/demo.pcap \
--label=data/demo.csv \
--output=out/OCSVM-results.dat

PCAP to features
To only extract features via the CLI:
netml learn extract \
--pcap=data/demo.pcap \
--label=data/demo.csv \
--feature=out/IAT-features.dat

Or in Python:
fromnetml.pparser.parserimportPCAPfromnetml.utils.toolimportdump_datapcap=PCAP(
'data/demo.pcap',
flow_ptks_thres=2,
random_state=42,
verbose=10,
)
# extract flows from pcappcap.pcap2flows(q_interval=0.9)
# label each flow (optional)pcap.label_flows(label_file='data/demo.csv')
# extract features from each flow via IATpcap.flow2features('IAT', fft=False, header=False)
# dump data to diskdump_data((pcap.features, pcap.labels), out_file='out/IAT-features.dat')
# statsprint(pcap.features.shape, pcap.pcap2flows.tot_time, pcap.flow2features.tot_time)
Features to model
To train from already-extracted features via the CLI:
netml learn train \
--feature=out/IAT-features.dat \
--output=out/OCSVM-results.dat

Or in Python:
fromsklearn.model_selectionimporttrain_test_splitfromnetml.ndm.modelimportMODELfromnetml.ndm.ocsvmimportOCSVMfromnetml.utils.toolimportdump_data, load_dataRANDOM_STATE=42# load data
(features, labels) =load_data('out/IAT-features.dat')
# split train and test sets
(
features_train,
features_test,
labels_train,
labels_test,
) =train_test_split(features, labels, test_size=0.33, random_state=RANDOM_STATE)
# create detection modelocsvm=OCSVM(kernel='rbf', nu=0.5, random_state=RANDOM_STATE)
ocsvm.name='OCSVM'ndm=MODEL(ocsvm, score_metric='auc', verbose=10, random_state=RANDOM_STATE)
# train the model from the train setndm.train(features_train)
# evaluate the trained modelndm.test(features_test, labels_test)
# dump data to diskdump_data((ocsvm, ndm.history), out_file='out/OCSVM-results.dat')
# statsprint(ndm.train.tot_time, ndm.test.tot_time, ndm.score)
For more examples, see the examples/ directory in the source repository.
Architecture
  • examples/
    
example code and datasets
  • src/netml/ndm/
    
detection models (such as OCSVM)
  • src/netml/pparser/
    
pcap processing (feature extraction)
  • src/netml/utils/
    
common functions (such as load_data and dump_data)
  • tests/
    
test cases
  • LICENSE.txt
  • manage.py
    
library development & management module
  • README.md
  • setup.cfg
  • setup.py
  • tox.ini
To Do
Further work includes:
  • Evaluate pparser performance on different pcaps
  • Add test cases
  • Add examples
  • Add (generated) docs
We welcome any comments to make this tool more robust and easier to use!
Development
Development dependencies may be installed via the dev extras (below assuming a source checkout):
pip install --editable .[dev]

(Note: the installation flag --editable is also used above to instruct pip to place the source checkout directory itself onto the Python path, to ensure that any changes to the source are reflected in Python imports.)
Development tasks are then managed via argcmdr sub-commands of manage …, (as defined by the repository module manage.py), e.g.:
manage version patch -m "initial release of netml" \
--build \
--release

Thanks
netml is based on the initial work of the "Outlier Detection" library odet 🙌
About
Network anomaly detection via machine learning
Resources
Readme
License
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages