MLFLow and scikit-multiflow, a good match

1. Generate abrupt drifts with scikit-multiflow (by changing the centroids in a Radial Basis Function stream)
2. Send this data, containing drifts, into a kafka topic
3. Listen to this kafka topic on another entry_point
4. Detect drifts - again with scikit-multiflow - on this entrypoint and log them into MLFlow

Steps 1. and 2. are done in producer.py, while steps 3. and 4. are implemented in detector.py

producer.py is just a script to generate synthetic data. Real-life applications should only require something looking like detector.py

Install & Setup (OSX)

For OSX users:

brew install librdkafka
brew cask install homebrew/cask-versions/adoptopenjdk8
brew install kafka

Creating topics with kafka

kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic drift_detection

Start zookeeper and kafka

brew services start zookeeper # TODO zookeeper should only be required by confluent-kafka, REMOVE ME ?
brew services start kafka

small issues, big impact

You need to add conda-forge to install packages like scikit-multiflow directly from conda

conda config --add topics conda-forge

RUN

First make the producer send data though kafka Here we generate 100 concept drift, pushing 10 rows into kafka every 0.5 seconds

mlflow run https://github.com/Quantmetry/mlflow.git/examples/abrupt_drift_detection -e producer -P n_drifts=100 -P interval=0.5 -P batch_size=10 -P n_classes=3

Then we have the detector listening on the right kafka topic, crashing logs whenever a drift is detected

mlflow run https://github.com/Quantmetry/MLflow_example.git -e detect_drifts -P delta=0.001

Finally visualize all this in the MLFLow UI

mlfow ui

Deleting topics with kafka, (to rerun failed experiments)

kafka-server-start /usr/local/etc/kafka/server.properties --override delete.topic.enable=true
kafka-topics --zookeeper 127.0.0.1:2181 --delete --topic drift_detection

Further readings

• Putting ML in productions using Kafka
• Adaptive Windowing