OSDroid

Predict running workflows' actions. Currently running at vocms0116.

Project under CMS CompOps T&I

How to run

After environments set up and necessary configuration pieces added, set up a crontab task to run periodically main.py.

To start the frontend instance,

cd web/
./quickStart.sh # this starts Flask basic server on port 8020 (subject to change)

Configurations etc.

A few more configuration files are needed to get it rolling.

1. config/config.yml for connections to UNIFIED DB, MySQL DB and alert email sending.

   oracle:
   - *** # username- *** # password- *** # db namemysql:
   - *** # username- *** # password- *** # db namealert_recipients:
   - XXX@YYYY.ZZ

2. config/credential.yml for stompAMQ to produce docs and authentication.

   producer: toolsandint-workflows-collectortopic: /topic/cms.toolsandint.workflowsinfocert: PATH_TO_CERT_FILE (***.pem)key: PATH_TO_KEY_FILE (***.rsa)jiracookie: PATH_TO_COOKIE_FILEhostport:
   host: XXXX.cern.chport: XXXXX

3. models/xgb_optimized.model for running workflow inference.

4. OSDroidDB is the local MySQL database storing workflow prediction history, labels and short-term document archive. Three tables need to be created for each.

   PredictionHistory

   CREATETABLEIF NOT EXISTS OSDroidDB.PredictionHistory (
   hid BIGINTNOT NULL AUTO_INCREMENT PRIMARY KEY,
   name VARCHAR(255) NOT NULL,
   good FLOAT,
   acdc FLOAT,
   resubmit FLOAT,
   timestampTIMESTAMP
   );

   LabelArchive

   CREATETABLEIF NOT EXISTS OSDroidDB.LabelArchive (
   name VARCHAR(255) NOT NULLPRIMARY KEY,
   label INT
   );

   DocsOneMonthArchive

   CREATETABLEIF NOT EXISTS OSDroidDB.DocsOneMonthArchive (
   id INTNOT NULL AUTO_INCREMENT PRIMARY KEY,
   name VARCHAR(255) NOT NULL,
   document LONGTEXT,
   timestampTIMESTAMP DEFAULT CURRENT_TIMESTAMP
   );

---

Data source

• UNIFIED DB
• wmstats server
• couchdb/acdc server
• ReqMgr2

Data storage

• stompAMQ (wrapped in CMSMonitoring) -> HDFS /CERN MONIT infrastructure

Model training

• pyspark (data fetch)
• SWAN
• XGBoost

Web SPA

• Flask
• dataTables.js
• plotly.js