11 statefun application - #18

Draft

Jungerindy wants to merge 268 commits into

mainfrom

11-statefun-application

This pull request is big! We're only showing the most recent 250 commits

Commits on Jun 4, 2024

adapting to statefun for high loads
Jungerindy
committed
932e833

Commits on Jun 6, 2024

small adaptions for kafkaProducer
Jungerindy
committed
9cdd0ab

Commits on Jun 10, 2024

adapt CPU amount for statefun runtime
Jungerindy
committed
817225d
adapt pred function for high load
Jungerindy
committed
25b1fe1

Commits on Jun 19, 2024

add env-vars to pred-functions
Jungerindy
committed
6fa9987
add env-vars to train-functions
Jungerindy
committed
5285965
Update README.md
Jungerindy
authored
2c4f7aa
pred functions code cleanup
Jungerindy
committed
6e636e4
train functions code cleanup
Jungerindy
committed
e4c5755
pred functions code cleanup
Jungerindy
committed
13b6565
Merge branch '11-statefun-application' of https://github.com/DPS-Stream-Processing-Applications/beam-applications-java into 11-statefun-application
Jungerindy
committed
e367762

Commits on Jun 26, 2024

trying to setupt latency tracking for statefun app
Jungerindy
committed
ccda6e3

Commits on Jun 28, 2024

Commits on Jul 22, 2024

adapt CMD command in dockerfiles
Jungerindy
committed
3e4a496
utils script for portforwarding
Jungerindy
committed
f2f9bef
custom latency tracking
Jungerindy
committed
ee69adc
small adaptions to statefun pred and train
Jungerindy
committed
7ccb122

Commits on Jul 23, 2024

change kafka topics
Jungerindy
committed
bfddcd1
logback for pred
Jungerindy
committed
bbcf5a7

Commits on Jul 26, 2024

changes in statefun functions
Jungerindy
committed
7c6e2ae

Commits on Jul 28, 2024

custom logging for functions
Jungerindy
committed
0d4b096
add latency logging to pred statefun
Jungerindy
committed
de60e42

Commits on Jul 30, 2024

Update README.md
Jungerindy
authored
f5c49cc

Commits on Aug 1, 2024

Commits on Aug 5, 2024

enable latency tracking for statefun
Jungerindy
committed
a8e29a2
add buffer debloat for statefun
Jungerindy
committed
b7d4f1a

Commits on Aug 6, 2024

disable custom logging
Jungerindy
committed
d6cc7dc
increase undertwo resouces
Jungerindy
committed
ecb2feb

Commits on Aug 20, 2024

add version to statefun-image
Jungerindy
committed
7b97da4

Commits on Aug 21, 2024

remove undertow server adjustments
Jungerindy
committed
97f8b7a
first version of statefunStarter
Jungerindy
committed
20c6ab1

Commits on Aug 22, 2024

trying to fix statefun crash
Jungerindy
committed
6add5e4

Commits on Aug 23, 2024

add functions-service to statefunStarter
Jungerindy
committed
6ed0fc6
statefunStarter implement setup of functions pod
Jungerindy
committed
bdadf7b

Commits on Aug 25, 2024

implement sending of kafka messages for statefunStarter
Jungerindy
committed
e15bc44
add proper topic forward-logic to statefunStarter
Jungerindy
committed
5ce062f

Commits on Aug 26, 2024

working on getting data from prometheus for statefunStarter
Jungerindy
committed
81403d1
adapt comparison numRecordsIn==numRecordsOut
Jungerindy
committed
8e79da3

Commits on Aug 27, 2024

remove kafka-publish with kafka-egress
Jungerindy
committed
913144a

Commits on Aug 28, 2024

add egress to module.yaml
Jungerindy
committed
ccf029f
adapt mqttPublish-func for train
Jungerindy
committed
58549d6
adapt resources for statefun-runtime
Jungerindy
committed
cf86e56
add prometheus reading to statefunStarter
Jungerindy
committed
69ec313

Commits on Aug 29, 2024

make improvements in statefunStarter to run on chameleon cluster
Jungerindy
committed
f539425

Commits on Aug 30, 2024

Commits on Sep 2, 2024

adaptions for statefunStarter.py
Jungerindy
committed
09319b6
adapt README for statefunStarter
Jungerindy
committed
7074431

Commits on Sep 8, 2024

adapt code for statefun functions
Jungerindy
committed
0e9541e

Commits on Sep 10, 2024

new limits for statefun-pods
Jungerindy
committed
0ac55cb

Commits on Sep 21, 2024

resource requests adaptions for statefun
Jungerindy
committed
eb41c8b

Commits on Sep 23, 2024

use protobuf serialization for pred functions
Jungerindy
committed
dc46a95

Commits on Sep 25, 2024

first draft of the scheduler
Jungerindy
committed
1110348

Commits on Sep 26, 2024

add serverless functionality to the scheduler prototype
Jungerindy
committed
98e37cd

Commits on Sep 27, 2024

Commits on Sep 28, 2024

adapt scheduler logic
Jungerindy
committed
13be505

Commits on Oct 4, 2024

add custom latency logging for train and pred statefun
Jungerindy
committed
a72a56f

Commits on Oct 5, 2024

first draft for timeseries forecasting
Jungerindy
committed
4c2c4cf

Commits on Oct 13, 2024

try new approach for time series prediction
Jungerindy
committed
fd6ddbe
adaptions in pred functions
Jungerindy
committed
1cdd731
adaptions in train functions
Jungerindy
committed
d317e0f

Commits on Oct 14, 2024

adapt statefun version
Jungerindy
committed
50950f8

Commits on Oct 15, 2024

add functioning prom reporter to statefun
Jungerindy
committed
68a0f17

Commits on Oct 16, 2024

prometheus script for getting metrics
Jungerindy
committed
81ef4bc

Commits on Oct 17, 2024

more generic model selection for timeseriesPredictor
Jungerindy
committed
b89c170

Commits on Oct 21, 2024

first implementation of scheduler logic
Jungerindy
committed
f2c0386

Commits on Oct 22, 2024

new formula for utility function
Jungerindy
committed
90df64e

Commits on Oct 23, 2024

add more metrics to time series basic example
Jungerindy
committed
58519af

Commits on Oct 25, 2024

restructuring scheduler directory
Jungerindy
committed
7dd2478
src-code structure for scheduler
Jungerindy
committed
9a7631e
scheduler config directory
Jungerindy
committed
a4db2bc
scheduler config directory
Jungerindy
committed
7f5d73a
initial structure of scheduler application
Jungerindy
committed
d7a7cad

Commits on Oct 26, 2024

add class for db access
Jungerindy
committed
d6162f6
add timeout logic
Jungerindy
committed
bdda960
scheduler_logic uses enums for frameworks
Jungerindy
committed
c363c59
create enums for frameworks and using two threads
Jungerindy
committed
bfc1cec
use enums in framework_scheduler
Jungerindy
committed
8ec6e39
create utils class with basic functions
Jungerindy
committed
4d3dbc5
remove class-design for db access
Jungerindy
committed
8b58f10
add event flag to scheduler logic
Jungerindy
committed
784b9d8
correct imports
Jungerindy
committed
d547f65
adapt imports
Jungerindy
committed
a8a29f3
add types to function arguments
Jungerindy
committed
14638f3
add helper method for quickly getting opposite framework
Jungerindy
committed
2f6bc81
adapt types for main.py
Jungerindy
committed
709acd7
skeleton for metrics_collector
Jungerindy
committed
df40b1d

Commits on Oct 27, 2024

add pytest cache to gitignore
Jungerindy
committed
4ca54ba
fix wrong variable name
Jungerindy
committed
af7784b
write test for components
Jungerindy
committed
4dbad29
remove debug code from main.py
Jungerindy
committed
8c2d71e
adapt kubernetes_service access
Jungerindy
committed
230a338
make kubernetes_service a script rather than a class
Jungerindy
committed
4d5db07
reformatting Utils.py
Jungerindy
committed
bffe75a
divide main_loop into two seperate methods
Jungerindy
committed
df51586
add interface of new function
Jungerindy
committed
a631e1e
test for framework_scheduler
Jungerindy
committed
cee0f5d

Commits on Oct 28, 2024

correct error in test case
Jungerindy
committed
5de684a
test evaluation monitor
Jungerindy
committed
4eba31d
test k8s service
Jungerindy
committed
19ddc2b
adapt timeout logic in evaluation monitor class
Jungerindy
committed
1f7a889
test for evaluation monitor
Jungerindy
committed
801bf41
add application and dataset as parameter
Jungerindy
committed
effdf48
adapt thread initialization
Jungerindy
committed
229e669
add methods for metric gathering
Jungerindy
committed
2795af8

Commits on Oct 29, 2024

write file for mysql access
Jungerindy
committed
fdbd4eb
write new tests
Jungerindy
committed
c9b9aa8
add mysql-connector to requirements
Jungerindy
committed
1e35674
implement check_safety net for evaluation monitor
Jungerindy
committed
8e7b82c
add timestamps to database access methods
Jungerindy
committed
4b71e5a

Commits on Oct 30, 2024

change functions for prometheus access
Jungerindy
committed
95eca69
add get_latency method for sl
Jungerindy
committed
fae3d9b
add entropy methods
Jungerindy
committed
178c358
add tests for entropy methods
Jungerindy
committed
c5efdc7

Commits on Oct 31, 2024

add tests for change method
Jungerindy
committed
f95a62f
add mock methods for db access
Jungerindy
committed
c5b4006
adapt to new function headers
Jungerindy
committed
8318a78
add db init to main-class
Jungerindy
committed
6538567
remove FIXME
Jungerindy
committed
6caace5
formatting code
Jungerindy
committed
8c38071
add method for getting numRecordsOut
Jungerindy
committed
785a762
remove comment
Jungerindy
committed
09ca056

Commits on Nov 1, 2024

add db env variables to manifest
Jungerindy
committed
30bd458
adapt Dockerfile to new file structure
Jungerindy
committed
54c2b80
adapt Makefile for local image
Jungerindy
committed
f1046ea
change new paths and add is_running flag
Jungerindy
committed
f75559e
add threshold dicts to main
Jungerindy
committed
c29335f
formatting
Jungerindy
committed
6c57aa0
fix typo in metricname
Jungerindy
committed
4173913
comment for local deployment
Jungerindy
committed
3c6fb8d

Commits on Nov 2, 2024

db methods for load prediction
Jungerindy
committed
834beef
add framework running flag
Jungerindy
committed
b0de60a
add paths for jars
Jungerindy
committed
9a59362
add framework running flag
Jungerindy
committed
e64180d
add simple ARIMA model
Jungerindy
committed
5bca24a
add framework running flag to main
Jungerindy
committed
f38c13c

Commits on Nov 3, 2024

add dtw similartiy calculation
Jungerindy
committed
b674005
adapt test to new event
Jungerindy
committed
292e8fb
add fast dtw requirement
Jungerindy
committed
02fb155

Commits on Nov 4, 2024

adapt db access class
Jungerindy
committed
fa76934
metrics collection for numRecordsIn
Jungerindy
committed
fe115dd
add window size for dtw
Jungerindy
committed
c40dd78
add new argument for evaluation monitor
Jungerindy
committed
24da183
switch to ARIMA
Jungerindy
committed
29114be
adapt test to changes
Jungerindy
committed
39c0774

Commits on Nov 5, 2024

adapt metric table to have input rate as column
Jungerindy
committed
9eb16a8
adapt metrics_collector to have method for getting input rate
Jungerindy
committed
67db71a
add inpute rate to evaluation monitor
Jungerindy
committed
aeeb70a
write change logic
Jungerindy
committed
7218836
try catch for ARIMA
Jungerindy
committed
9cf5fc2
add main to similartiy calculation
Jungerindy
committed
8d03702
change test to support input rate
Jungerindy
committed
3ab1be5
test make_change function
Jungerindy
committed
1e94b6b
fix formatting
Jungerindy
committed
e2872bb
add penalty for not running framework
Jungerindy
committed
7fac488
remove SARIMAX methods
Jungerindy
committed
16549d7
remove SARIMAX methods
Jungerindy
committed
a294ae2
adapt debug method
Jungerindy
committed
0060f27
change np version for statsmodels
Jungerindy
committed
17afddd
write second test for run_evaluation
Jungerindy
committed
4804eb2
skip test for SARIMAX
Jungerindy
committed
d8323b7

Commits on Nov 7, 2024

add comments to main class
Jungerindy
committed
0cac268
update README with mysql setup
Jungerindy
committed
5e2fc92
add assert for sent messages
Jungerindy
committed
788c8e7
add return statement to main_loop_logic
Jungerindy
committed
a1aee30
add main method for debugging
Jungerindy
committed
ed41d54
remove commented out lines
Jungerindy
committed
d12f62f
remove comments from Dockerfile
Jungerindy
committed
56ca84d
add stacktrace in case of errors
Jungerindy
committed
1a5fec1
adapt metrics class to sf and an try/catch
Jungerindy
committed
61769f3
add test for None to evaluation_monitor
Jungerindy
committed
0b8e4e5
add logging statements for db methods
Jungerindy
committed
601a627
add fail safe for k8s_service numRecordsOut
Jungerindy
committed
5ccdd27
correct error message for metric method
Jungerindy
committed
2854362
add fail safe if no metrics can be scraped
Jungerindy
committed
a6ea298
add check for lenght of historic data before predicting time series
Jungerindy
committed
f964526
clean up imports
Jungerindy
committed
b4da8e8
add additional logs
Jungerindy
committed
c59e0fa
add test data
Jungerindy
committed
3dac7b9
add kafka-topic manifest
Jungerindy
committed
c05a7bf
add command to log to show everything
Jungerindy
committed
bb57e58
adapt README to new folder structure
Jungerindy
committed
738761b
add pod_monitors for prometheus
Jungerindy
committed
60230be
change strings for framework enums
Jungerindy
committed
6e3d246
adapt check validity function
Jungerindy
committed
d3bc665

Commits on Nov 8, 2024

change level of logs
Jungerindy
committed
ba34440
exit main_loop if setup fails
Jungerindy
committed
d32f39f
adapt paths to new folder structure
Jungerindy
committed
d98258c
handle cases for None value
Jungerindy
committed
20740bb
add logging for metrics
Jungerindy
committed
f1c0946
add logic to check length of ARIMA train data
Jungerindy
committed
07ecc3e
add returns in error case
Jungerindy
committed
b5bf20f
comment out print statements
Jungerindy
committed
944ea5d
add new framework manifest for sl to container
Jungerindy
committed
51a676a
add docker command to Makefile
Jungerindy
committed
222915f

Commits on Nov 9, 2024

access kafka message with .value to allow decoding and encoding
Jungerindy
committed
b2ddede
adapt check_operator_name_sf for critical sf_metrics
Jungerindy
committed
d63d455
add targets to make to build and push dockerimage to dockerhub
Jungerindy
committed
ca7f0d8
fix get latency_sl
Jungerindy
committed
fc6e10f
change logging level
Jungerindy
committed
574aeb4
change error case for numRecordsOut_sl
Jungerindy
committed
fdf9568
format code
Jungerindy
committed
45da647
new content for README
Jungerindy
committed
b7e1005

Commits on Nov 10, 2024

add debug flag
Jungerindy
committed
31a3cc1
integreate debug flag
Jungerindy
committed
0890bd5

Commits on Nov 11, 2024

change imagepull strategy for CC running
Jungerindy
committed
363ec3b
add pmdarima
Jungerindy
committed
edc6e60
add method to retrieve input rates after certain timestamp
Jungerindy
committed
1583c15
make kafkaconsumer for latency_sl singleton
Jungerindy
committed
ef932ef
add new auto_arima instance for evaluation_monitor
Jungerindy
committed
81835b0
implement new logic for auto_arima in scheduler_logic
Jungerindy
committed
7fdb8f8
add methods for auto_arima and attributes for class logic
Jungerindy
committed
f11a7cf
adapt test to changes for auto_arima implementation
Jungerindy
committed
27c6dbd
adapt get_latency_values_sf to new messages
Jungerindy
committed
9f04eca
write test for get_latency_sf
Jungerindy
committed
45f74e4
add timestamp to sf latency
Jungerindy
committed
3743221

Commits on Nov 13, 2024

add file for port-forwarding db-access
Jungerindy
committed
520ff01
remove debug flags for db
Jungerindy
committed
147e9b4
fix method for retrieving input rates
Jungerindy
committed
6fe9150
remove FIXME comment
Jungerindy
committed
bfbdab8
use new retrieve-db method
Jungerindy
committed
7afee6f
adapt tests
Jungerindy
committed
8beaee0
dummy test for db_access
Jungerindy
committed
0bd3eed

Commits on Nov 16, 2024

change to correct database name
Jungerindy
committed
e286859
add cluster roles to README instructions
Jungerindy
committed
d49c490
adapt test to code changes
Jungerindy
committed
fb2541a
adapt user parameters
Jungerindy
committed
f045586
add new column to decision table
Jungerindy
committed
b74a177
adapt store_decision call
Jungerindy
committed
6b3f59d

Commits on Nov 18, 2024

add env variable for window size
Jungerindy
committed
db948ce
adapt documentation of delete_tables method
Jungerindy
committed
a37ca3b
change values and naming of sleeping intervals for monitoring
Jungerindy
committed
62e40b8
add logging for selected window size
Jungerindy
committed
a8d8359
adapt tests to new naming for evaluation_monitor
Jungerindy
committed
c6ac52e
new latency logging for pred statefunApplication
Jungerindy
committed
c194d3f
add test for make_decision method
Jungerindy
committed
bd9e44d
make correct doc strings for db methods
Jungerindy
committed
0b2ce3f
implement threshold logic instead of penalty
Jungerindy
committed
08be9ac

Commits on Nov 19, 2024

Commits on Nov 21, 2024

add method for retrieving input rate with exact timestamp
Jungerindy
committed
e2fb600
implement handle in case ARIMA has only constant input rates
Jungerindy
committed
04cd648
add test case for constant input to ARIMA
Jungerindy
committed
9caec31