Uh oh!
There was an error while loading. Please reload this page.
This repository was archived by the owner on Sep 12, 2024. It is now read-only.
- Notifications
You must be signed in to change notification settings - Fork 51
Python3 Migration Notes
Amanda Goldani edited this page Aug 13, 2021
·
9 revisions
This page is intended to show the differences between the old version of the code and the new one.
Note that all Python3 top levels are located in src/python/, this was omitted in the details for convenience.
Includes all cache functions and classes from utils.py.
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/cacheInfo.from_file | Cache/CacheManager.py/CacheManager._getFromFile | ✅ | * called in get |
utils.py/cacheInfo._file_key | Cache/CacheManager.py/CacheManager._getKeyFilePath | ✅ | * called in get |
utils.py/cacheInfo.get | Cache/CacheManager.py/CacheManager.get | ✅ | |
utils.py/cacheInfo.purge | Cache/CacheManager.py/CacheManager.purge | ❌ | |
utils.py/cacheInfo.store | Cache/CacheManager.py/CacheManager.set | ❌ | |
utils.py/docCache.get | Cache/DataCacheLoader.py/DataCacheLoader.get | ❌ | |
utils.py/docCache.__init__.cache | Cache/DocumentCache.py | ❌ | * each possible document cache key became a class with the corresponding get method. The DataCacheLoader makes the mapping and loads the document data when required |
| — | Cache/CacheManager.py/CacheManager._buildMongoDocument | ✅ | * called in set |
| — | Cache/CacheManager.py/CacheManager._setMongoCollection | ✅ | * required by the MongoClient |
Includes the helper function from utils.py that creates the connection to MongoDB, as well as the migration of the file that creates the connection to OracleDB.
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/mongo_client | Databases/Mongo/MongoClient.py | — | * this is an abstract class where the mongo collection must be defined in the child classes |
Unified/assignSession.py | Databases/Oracle/OracleClient.py | ✅ | * tested through the OracleDB test file |
Unified/assignSchema.py | Databases/Oracle/OracleDB.py | ✅ |
Includes all the helper classes from utils.py controlling a MongoDB collection.
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/agentInfoDB | MongoControllers/AgentController.py/AgentController._setMongoCollection | ✅ | * required by the MongoClient |
utils.py/agentInfo.agentStatus | MongoControllers/AgentController.py/AgentController.get | ✅ | * this will return all the agent's data, use result.get("status") to keep the same behavior |
utils.py/agentInfo.speed_draining | MongoControllers/AgentController.py/AgentController.getAgents(speeddrain=True) | ✅ | |
utils.py/agentInfo.poll | MongoControllers/AgentController.py/AgentController.poll | ❌ | |
utils.py/agentInfo.change_status | MongoControllers/AgentController.py/AgentController.set | ❌ | |
utils.py/agentInfo.getStatus | MongoControllers/AgentController.py/AgentController.syncToProduction | ❌ | |
utils.py/agentInfo.checkTrello | MongoControllers/AgentController.py/AgentController.syncToTrelloMongoControllers/AgentController.py/AgentController.updateTrelloBoard | ❌ | * split because the Trello board was updated or the collection was sync to the board (never both!) |
| — | MongoControllers/AgentController.py/AgentController._areRunningOverThreshold | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._bootAgent | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._buildMongoDocument | ✅ | * called in set |
| — | MongoControllers/AgentController.py/AgentController._countAgentsWithReleaseStatus | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._countJobsAndCpus | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._drainAgent | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._initializePollParams | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._isLight | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._isStuffed | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._newRelease | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._retireAgent | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._setCandidatesForDraining | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._setCandidatesForOpenDraining | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._setCandidatesForSpeedDraining | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._setCandidatesForWakingUp | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._setLastestAndOldestReleases | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._setRecentAgents | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._setSpeedDrainingPriority | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._sumRunningCapacity | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._syncAgentStatus | ❌ | * called in syncToProduction |
| — | MongoControllers/AgentController.py/AgentController._tell | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController._updateAgentConfig | ❌ | * called in poll |
| — | MongoControllers/AgentController.py/AgentController.getAgents | ✅ |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/batchInfo.content | MongoControllers/BatchController.py/BatchController.get | ✅ | |
utils.py/batchInfo.all | MongoControllers/BatchController.py/BatchController.getBatches | ✅ | |
utils.py/batchInfo.pop | MongoControllers/BatchController.py/BatchController.pop | ❌ | |
utils.py/batchInfo.updateutils.py/batchInfo.add | MongoControllers/BatchController.py/BatchController.set | ❌ | * unified because add is only called inside update |
| — | MongoControllers/BatchController.py/BatchController._buildMongoDocument | ✅ | * called in set |
| — | MongoControllers/BatchController.py/BatchController._setMongoCollection | ✅ | * required by the MongoClient |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/campaignInfo.content | MongoControllers/CampaignController.py/CampaignController.get | ✅ | |
utils.py/campaignInfo.all | MongoControllers/CampaignController.py/CampaignController.getCampaigns | ✅ | |
utils.py/campaignInfo.parameters | MongoControllers/CampaignController.py/CampaignController.getCampaignParameters | ✅ | |
utils.py/campaignInfo.allSecondaries | MongoControllers/CampaignController.py/CampaignController.getSecondaries | ✅ | |
utils.py/campaignInfo.get | MongoControllers/CampaignController.py/CampaignController.getCampaignValue | ✅ | |
utils.py/campaignInfo.go | MongoControllers/CampaignController.py/CampaignController.go | ✅ | |
utils.py/campaignInfo.pop | MongoControllers/CampaignController.py/CampaignController.pop | ❌ | |
utils.py/campaignInfo.updateutils.py/campaignInfo.add | MongoControllers/CampaignController.py/CampaignController.set | ❌ | * unified because add is only called inside update |
| — | MongoControllers/CampaignController.py/CampaignController._buildMongoDocument | ✅ | * called in set |
| — | MongoControllers/CampaignController.py/CampaignController._setCampaigns | ✅ | * called when instantiated |
| — | MongoControllers/CampaignController.py/CampaignController._setMongoCollection | ✅ | * required by the MongoClient |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/closeoutInfo.one_line | MongoControllers/CloseoutController.py/CloseoutController._buildHtmlRow | ❌ | * called in assistance and summary |
utils.py/closeoutInfo.assistance | MongoControllers/CloseoutController.py/CloseoutController.assistance | ❌ | |
utils.py/closeoutInfo.pop | MongoControllers/CloseoutController.py/CloseoutController.clean | ❌ | |
utils.py/closeoutInfo.get | MongoControllers/CloseoutController.py/CloseoutController.get | ✅ | |
utils.py/closeoutInfo.html | MongoControllers/CloseoutController.py/CloseoutController.html | ❌ | |
utils.py/closeoutInfo.update | MongoControllers/CloseoutController.py/CloseoutController.set | ❌ | |
utils.py/closeoutInfo.summary | MongoControllers/CloseoutController.py/CloseoutController.summary | ❌ | |
utils.py/closeoutInfo.table_header | Included in the CloseoutController templates | — | |
| — | MongoControllers/CloseoutController.py/CloseoutController._buildMongoDocument | ✅ | * called in set |
| — | MongoControllers/CloseoutController.py/CloseoutController._renderAssistanceHtml | ❌ | * called in assistance |
| — | MongoControllers/CloseoutController.py/CloseoutController._renderSummaryHtml | ❌ | * called in summary |
| — | MongoControllers/CloseoutController.py/CloseoutController._setMongoCollection | ✅ | * required by the MongoClient |
| — | MongoControllers/CloseoutController.py/CloseoutController.getWorkflows | ✅ |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/moduleLock.check | MongoControllers/ModuleLockController.py/ModuleLockController.check | ❌ | |
utils.py/moduleLock.clean | MongoControllers/ModuleLockController.py/ModuleLockController.clean | ❌ | |
utils.py/moduleLock.all_locks | MongoControllers/ModuleLockController.py/ModuleLockController.get | ✅ | |
| — | MongoControllers/ModuleLockController.py/ModuleLockController._buildMongoDocument | ✅ | * called in set |
| — | MongoControllers/ModuleLockController.py/ModuleLockController._setMongoCollection | ✅ | * required by the MongoClient |
| — | MongoControllers/ModuleLockController.py/ModuleLockController.go | ✅ | |
| — | MongoControllers/ModuleLockController.py/ModuleLockController.pop | ❌ | |
| — | MongoControllers/ModuleLockController.py/ModuleLockController.set | ❌ |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/remainingDatasetInfo.clean | MongoControllers/RemainingDatasetController.py/RemainingDatasetController.clean | ❌ | |
utils.py/remainingDatasetInfo.get | MongoControllers/RemainingDatasetController.py/RemainingDatasetController.get | ✅ | |
utils.py/remainingDatasetInfo.sites | MongoControllers/RemainingDatasetController.py/RemainingDatasetController.getSites | ✅ | |
utils.py/remainingDatasetInfo.purge | MongoControllers/RemainingDatasetController.py/RemainingDatasetController.purge | ❌ | |
utils.py/remainingDatasetInfo.set | MongoControllers/RemainingDatasetController.py/RemainingDatasetController.set | ❌ | |
utils.py/remainingDatasetInfo.sync | MongoControllers/RemainingDatasetController.py/RemainingDatasetController.syncMongoControllers/RemainingDatasetController.py/RemainingDatasetController.syncSite | ❌ | * split to let explicit if one site or all sites are being sync |
utils.py/remainingDatasetInfo.tell | MongoControllers/RemainingDatasetController.py/RemainingDatasetController.tell | ✅ | |
| — | MongoControllers/RemainingDatasetController.py/RemainingDatasetController._buildMongoDocument | ✅ | * called in set |
| — | MongoControllers/RemainingDatasetController.py/RemainingDatasetController._setMongoCollection | ✅ | * required by the MongoClient |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/reportInfo._convert | MongoControllers/ReportController.py/ReportController._convertValues | ✅ | * called in set |
utils.py/reportInfo.get | MongoControllers/ReportController .py/ReportController.get | ✅ | |
utils.py/reportInfo.purge | MongoControllers/ReportController .py/ReportController.purgeMongoControllers/ReportController .py/ReportController.cleanMongoControllers/ReportController .py/ReportController.cleanWorkflow | ❌ | * split because the old function would do different things depending os the input params |
utils.py/reportInfo._put | MongoControllers/ReportController.py/ReportController.set | ❌ | |
utils.py/reportInfo.set_blocks | MongoControllers/ReportController.py/ReportController.setWorkflowBlocks | ❌ | |
utils.py/reportInfo.set_errors | MongoControllers/ReportController.py/ReportController.setWorkflowErrors | ❌ | |
utils.py/reportInfo.set_files | MongoControllers/ReportController.py/ReportController.setWorkflowFiles | ❌ | |
utils.py/reportInfo.set_IO | MongoControllers/ReportController.py/ReportController.setWorkflowIO | ❌ | |
utils.py/reportInfo.set_logs | MongoControllers/ReportController.py/ReportController.setWorkflowLogs | ❌ | |
utils.py/reportInfo.set_missing | MongoControllers/ReportController.py/ReportController.setWorkflowMissing | ❌ | |
utils.py/reportInfo._set_for_task | MongoControllers/ReportController.py/ReportController.setWorkflowTask | ❌ | |
utils.py/reportInfo.set_ufiles | MongoControllers/ReportController.py/ReportController.setWorkflowUFiles | ❌ | |
| — | MongoControllers/ReportController.py/ReportController._buildMongoDocument | ✅ | * called in set |
| — | MongoControllers/ReportController.py/ReportController._setMongoCollection | ✅ | * required by the MongoClient |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/StartStopInfo.get | MongoControllers/StartStopTimeController.py/StartStopTimeController.get | ✅ | |
utils.py/StartStopInfo.purge | MongoControllers/StartStopTimeController.py/StartStopTimeController.purge | ❌ | |
utils.py/StartStopInfo.pushStartStopTime | MongoControllers/StartStopTimeController.py/StartStopTimeController.set | ❌ | |
| — | MongoControllers/StartStopTimeController.py/StartStopTimeController._buildMongoDocument | ✅ | * called in set |
| — | MongoControllers/StartStopTimeController.py/StartStopTimeController._setMongoCollection | ✅ | * required by the MongoClient |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/statusHistory.py/statusHistory.content | MongoControllers/StatusHistoryController.py/StatusHistoryController.get | ✅ | |
utils.py/statusHistory.py/statusHistory.trim | MongoControllers/StatusHistoryController.py/StatusHistoryController.purge | ❌ | |
utils.py/statusHistory.py/statusHistory.add | MongoControllers/StatusHistoryController.py/StatusHistoryController.set | ❌ | |
| — | MongoControllers/StatusHistoryController.py/StatusHistoryController._buildMongoDocument | ✅ | * called in set |
| — | MongoControllers/StatusHistoryController.py/StatusHistoryController._setMongoCollection | ✅ | * required by the MongoClient |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/wtcInfo.clean | MongoControllers/WTCController.py/WTCController.clean | ❌ | |
utils.py/wtcInfo.remove | MongoControllers/WTCController.py/WTCController.cleanKeyword | ❌ | |
utils.py/wtcInfo._get | MongoControllers/WTCController.py/WTCController.get | ✅ | |
utils.py/wtcInfo.getBypass | MongoControllers/WTCController.py/WTCController.getBypass | ✅ | |
utils.py/wtcInfo.getForce | MongoControllers/WTCController.py/WTCController.getForce | ✅ | |
utils.py/wtcInfo.getHold | MongoControllers/WTCController.py/WTCController.getHold | ✅ | |
utils.py/wtcInfo.add | MongoControllers/WTCController.py/WTCController.set | ❌ | |
| — | MongoControllers/WTCController.py/WTCController._buildMongoDocument | ✅ | * called in set |
| — | MongoControllers/WTCController.py/WTCController._setMongoCollection | ✅ | * required by the MongoClient |
Includes all helper functions from utils.py talking to a service.
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/workflowInfo.getRecoveryDoc | Services/ACDC/ACDCReader.py/ACDCReader.getRecoveryDocs | ✅ | * input arg collection_name was switched by the workflow name because it was never used |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/componentCheck.check_dbs | Services/DBS/DBSReader.py/DBSReader.check | ❌ | |
utils.py/getFileBlock | Services/DBS/DBSReader.py/DBSReader.getBlockName | ✅ | |
utils.py/getDatasetBlocks | Services/DBS/DBSReader.py/DBSReader.getDatasetBlockNamesServices/DBS/DBSReader.py/DBSReader.getDatasetBlockNamesByLumisServices/DBS/DBSReader.py/DBSReader.getDatasetBlockNamesByRuns | ✅ | * split because the block names were filtered by runs or by lumis (never both!) |
utils.py/getDatasetEventsAndLumis | Services/DBS/DBSReader.py/DBSReader.getDatasetEventsAndLumisServices/DBS/DBSReader.py/DBSReader.getBlocksEventsAndLumis | ✅ | * split because events and lumis were got by dataset or blocks (never both!) |
utils.py/getDatasetEventsPerLumis | Services/DBS/DBSReader.py/DBSReader.getDatasetEventsPerLumis | ✅ | |
utils.py/getDatasetFileArray | Services/DBS/DBSReader.py/DBSReader.getDatasetFiles | ✅ | * input args use_array and cache_timeout were removed because were not used |
utils.py/getDatasetLumisAndFiles | Services/DBS/DBSReader.py/DBSReader.getDatasetLumisAndFiles | ✅ | * only does the request, input args runs and lumilist were removed but the helper functions Utilities/DataTools.py/filterLumisAndFilesByRuns or filterLumisAndFilesByLumis can be used if needed |
utils.py/getDatasets | Services/DBS/DBSReader.py/DBSReader.getDatasetNames | ✅ | |
utils.py/findParent | Services/DBS/DBSReader.py/DBSReader.getDatasetParent | ✅ | |
utils.py/getDatasetRuns | Services/DBS/DBSReader.py/DBSReader.getDatasetRuns | ✅ | |
utils.py/getDatasetSize | Services/DBS/DBSReader.py/DBSReader.getDatasetSize | ✅ | |
utils.py/getDatasetStatus | Services/DBS/DBSReader.py/DBSReader.getDBSStatus | ✅ | |
utils.py/getFilesWithLumiInRun | Services/DBS/DBSReader.py/DBSReader.getFilesWithLumiInRun | ✅ | |
utils.py/getLFNbase | Services/DBS/DBSReader.py/DBSReader.getLFNBase | ✅ | |
utils.py/setDatasetStatus | Services/DBS/DBSWriter.py/DBSWriter.setDatasetStatus | ❌ | |
utils.py/setFileStatus | Services/DBS/DBSWriter.py/DBSWriter.setFileStatus | ❌ | |
| — | Services/DBS/DBSReader.py/DBSReader._getBlockFileLumis | ❌ | * called in getBlocksLumisAndFilesForCaching |
| — | Services/DBS/DBSReader.py/DBSReader._getFileLumiArray | ❌ | * called in getFilesWithLumiInRun |
| — | Services/DBS/DBSReader.py/DBSReader.getBlocksLumisAndFilesForCaching | ✅ | |
| — | Services/DBS/DBSReader.py/DBSReader.getRecoveryBlocksAndLocations | ❌ |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/eosRead | Services/EOS/EOSReader.py/EOSReader.read | ✅ | |
utils.py/eosFile.close | Services/EOS/EOSWriter.py/EOSWriter.save | ❌ | |
utils.py/eosFile.write | Services/EOS/EOSWriter.py/EOSWriter.write | ❌ |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/workflowInfo.getGlideMon | Services/GWMSMon/GWMSMonReader.py/GWMSMonReader.getRequestSummary | ❌ | * request return an error |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/getAgentConfig | Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getAgentConfig | ❌ | |
utils.py/getReqmgrInfo | Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getReqmgrInfo | ✅ | |
utils.py/getWorkflowByCampaign | Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowsByCampaign | ✅ | |
utils.py/getWorkflowByName | Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowsByNames | ✅ | |
utils.py/getWorkflowByOutput | Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowsByOutput | ✅ | * endpoint changed from CouchDB to the RESTful api |
utils.py/getWorkflowById | Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowsByPrepId | ✅ | * endpoint changed from CouchDB to the RESTful api |
utils.py/getWorkflows | Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowsByStatus | ✅ | |
utils.py/getWorkload | Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowSchema | ✅ | |
utils.py/workflowInfo.getSchema | Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowSchema(makeCopy=True) | ✅ | |
utils.py/workflowInfo.getSummary | Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkloadSummary | ✅ | |
utils.py/workflowInfo.get_spec | Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getSpec | ✅ | |
utils.py/workflowInfo.getSplittingsNew | Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getSplittingsSchema | ✅ | * only does the request, input args strip and allTasks were removed but the helper functions Utilities/DataTools.py/filterSplittingsParam and filterSplittingsTaskTypes can be used if needed |
utils.py/sendAgentConfig | Services/ReqMgr/ReqMgrWriter.py/ReqMgrWriter.setAgentConfig | ❌ | |
| — | Services/ReqMgr/ReqMgrReader.py/ReqMgrReader.getWorkflowsByParam | ❌ |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
RucioClient.py/getFileCountBlock | Services/Rucio/RucioReader.py/RucioReader.countBlockFiles | ✅ | |
RucioClient.py/getFileCountDataset | Services/Rucio/RucioReader.py/RucioReader.countDatasetFiles | ✅ | |
RucioClient.py/getFileCountPerBlock | Services/Rucio/RucioReader.py/RucioReader.countDatasetFilesPerBlock | ✅ | |
RucioClient.py/getBlockNamesDataset | Services/Rucio/RucioReader.py/RucioReader.getDatasetBlockNames | ✅ | |
RucioClient.py/getFileNamesDataset | Services/Rucio/RucioReader.py/RucioReader.getDatasetFileNames | ✅ | |
RucioClient.py/getDatasetLocationsByAccount | Services/Rucio/RucioReader.py/RucioReader.getDatasetLocationsByAccount | ✅ |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
TrelloClient.py/_get | Services/Trello/TrelloClient.py/TrelloClient._getResponse | ❌ | * called in getBoard and getCard |
TrelloClient.py/_put | Services/Trello/TrelloClient.py/TrelloClient._sendResponse | ❌ | * called in setList |
TrelloClient.py/getAgents | Services/Trello/TrelloClient.py/TrelloClient._syncAgents | ❌ | * called when instantiated |
TrelloClient.py/getBoard | Services/Trello/TrelloClient.py/TrelloClient.getBoard | ❌ | |
TrelloClient.py/getCard | Services/Trello/TrelloClient.py/TrelloClient.getCard | ❌ | |
TrelloClient.py/changeList | Services/Trello/TrelloClient.py/TrelloClient.setList | ❌ | |
| — | Services/Trello/TrelloClient.py/TrelloClient.getListId | ❌ |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/getAllAgents | Services/WMStats/WMStatsReader.py/WMStatsReader.getAgents | ✅ | |
utils.py/workflowInfo.getWMErrors | Services/WMStats/WMStatsReader.py/WMStatsReader.getWMErrors | ❌ | * only include the request of the original function — call WorkflowController.getWMErrors to keep the same behavior |
utils.py/workflowInfo.getWMStats | Services/WMStats/WMStatsReader.py/WMStatsReader.getWMStats | ✅ | * only include the request of the original function — call WorkflowController.getWMStats to keep the same behavior |
| — | Services/WMStats/WMStatsReader.py/WMStatsReader.getProductionAgents | ✅ |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/workflowInfo.getWorkQueue | Services/WorkQueue/WorkQueueReader.py/WorkQueueReader.getWorkQueue | ✅ |
Includes all helper functions from utils.py that are used for authentication, configuration, etc.
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/make_x509_conn | Utilities/Authenticate.py/getX509Conn | ❌ | |
utils.py/unifiedConfiguration | Utilities/ConfigurationHandler.py/ConfigurationHandler | ❌ | |
utils.py/workflowInfo._taskDescending | Utilities/DataTools/flattenTaskTree | ❌ | |
utils.py/ThreadHandler | Utilities/Decorators.py/runWithMultiThreading | ❌ | |
utils.py/runWithRetries | Utilities/Decorators.py/runWithRetries | ❌ | |
utils.py/display_N | Utilities/Logging/displayNumber | ❌ | |
utils.py/display_time | Utilities/Logging/displayTime | ❌ | |
utils.py/GET | Utilities/WebTools.py/getResponse | ❌ | |
| — | Utilities/DataTools/filterFilesAndLocationsInDBS | ❌ | |
| — | Utilities/DataTools/filterLumisAndFilesByLumis | ❌ | |
| — | Utilities/DataTools/filterLumisAndFilesByRuns | ❌ | |
| — | Utilities/DataTools/filterRecoveryFilesAndLocations | ❌ | |
| — | Utilities/DataTools/filterSplittingsParam | ❌ | |
| — | Utilities/DataTools/filterSplittingsTaskTypes | ❌ | |
| — | Utilities/DataTools/sortByWakeUpPriority | ❌ | |
| — | Utilities/IteratorTools/filterKeys | ❌ | |
| — | Utilities/IteratorTools/mapKeys | ❌ | |
| — | Utilities/IteratorTools/mapValues | ❌ | |
| — | Utilities/IteratorTools/sortByKeys | ❌ | |
| — | Utilities/WebTools.py/sendResponse | ❌ |
Includes all helper functions from utils.py that are used for managing a workflow
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/workflowInfo.checkSplitting | WorkflowMgmt/WorkflowController.py/WorkflowController.checkSplitting | ✅ | |
utils.py/workflowInfo.getAgents | WorkflowMgmt/WorkflowController.py/WorkflowController.getAgents | ✅ | |
utils.py/workflowInfo.getAllTasks | WorkflowMgmt/WorkflowController.py/WorkflowController.getAllTasks | ✅ | |
utils.py/workflowInfo.getBlocks | WorkflowMgmt/WorkflowController.py/WorkflowController.getBlocks | ✅ | |
utils.py/workflowInfo.getBlockWhiteList | WorkflowMgmt/WorkflowController.py/WorkflowController.getBlockWhiteList | ✅ | |
utils.py/workflowInfo.getBlowupFactors | WorkflowMgmt/WorkflowController.py/WorkflowController.getBlowupFactor | ✅ | * return only the factor, since the additional outputs were not used |
utils.py/workflowInfo.getCampaignPerTask | WorkflowMgmt/WorkflowController.py/WorkflowController.getCampaignByTask | ✅ | |
utils.py/workflowInfo.getConfigCacheID | WorkflowMgmt/WorkflowController.py/WorkflowController.getConfigCacheID | ✅ | |
utils.py/workflowInfo.getCompletionFraction | WorkflowMgmt/WorkflowController.py/WorkflowController.getCompletionFraction | ✅ | |
utils.py/workflowInfo.getComputingTime | `WorkflowMgmt/WorkflowController.py/WorkflowController.getComputingTime | ✅ | |
utils.py/workflowInfo.getCorePerTask | WorkflowMgmt/WorkflowController.py/WorkflowController.getCoreByTask | ✅ | |
utils.py/workflowInfo.getFamilly | WorkflowMgmt/WorkflowController.py/WorkflowController.getFamily | ✅ | |
utils.py/workflowInfo.getFilterEfficiency | WorkflowMgmt/WorkflowController.py/WorkflowController.getFilterEfficiencyByTask | ✅ | |
utils.py/workflowInfo.firstTask | WorkflowMgmt/WorkflowController.py/WorkflowController.getFirstTask | ✅ | |
utils.py/workflowInfo.getGlideMon | WorkflowMgmt/WorkflowController.py/WorkflowController.getGlideWMSMonSummary | ❌ | |
utils.py/workflowInfo.getMemoryPerTask | WorkflowMgmt/WorkflowController.py/WorkflowController.getMemoryByTask | ✅ | |
utils.py/workflowInfo.getNCopies | WorkflowMgmt/WorkflowController.py/WorkflowController.getNCopies | ✅ | |
utils.py/workflowInfo.getNextVersion | WorkflowMgmt/WorkflowController.py/WorkflowController.getNextVersion | ✅ | |
utils.py/workflowInfo.getOutputPerTask | WorkflowMgmt/WorkflowController.py/WorkflowController.getOutputDatasetsPerTask | ✅ | |
utils.py/workflowInfo.getPrepIDs | WorkflowMgmt/WorkflowController.py/WorkflowController.getPrepIDs | ✅ | |
utils.py/workflowInfo.getRecoveryBlocks | WorkflowMgmt/WorkflowController.py/WorkflowController.getRecoveryBlocks | ❌ | |
utils.py/workflowInfo.getRecoveryInfo | WorkflowMgmt/WorkflowController.py/WorkflowController.getRecoveryInfo | ❌ | |
utils.py/workflowInfo.getRunWhiteList | WorkflowMgmt/WorkflowController.py/WorkflowController.getRunWhiteList | ✅ | |
utils.py/workflowInfo.getArchs | WorkflowMgmt/WorkflowController.py/WorkflowController.getScramArches | ✅ | |
utils.py/workflowInfo.getSiteWhiteList | WorkflowMgmt/WorkflowController.py/WorkflowController.getSiteWhiteList | ❌ | |
utils.py/workflowInfo.getSpplitings | WorkflowMgmt/WorkflowController.py/WorkflowController.getSpplitings | ✅ | |
utils.py/workflowInfo.getSplittingsNew | WorkflowMgmt/WorkflowController.py/WorkflowController.getSpplitingsSchema | ✅ | |
utils.py/workflowInfo.getSummary | WorkflowMgmt/WorkflowController.py/WorkflowController.getSummary | ✅ | |
utils.py/workflowInfo.getWMErrors | WorkflowMgmt/WorkflowController.py/WorkflowController.getWMErrors | ❌ | |
utils.py/workflowInfo.getWMStatus | WorkflowMgmt/WorkflowController.py/WorkflowController.getWMStatus | ✅ | |
utils.py/workflowInfo.getWorkTasks | WorkflowMgmt/WorkflowController.py/WorkflowController.getWorkTasks | ✅ | |
utils.py/workflowInfo.go | WorkflowMgmt/WorkflowController.py/WorkflowController.go | ✅ | |
utils.py/workflowInfo.heavyRead | WorkflowMgmt/WorkflowController.py/WorkflowController.isHeavyToReady | ✅ | |
| — | WorkflowMgmt/WorkflowController.py/WorkflowController._getAllowedSites | ❌ | * called in getSiteWhiteList |
| — | WorkflowMgmt/WorkflowController.py/WorkflowController._getFromFile | ✅ | * called in getWMStats and getWMErrors |
| — | WorkflowMgmt/WorkflowController.py/WorkflowController._getVersionByConflictingWorkflows | ✅ | * called in getNextVersion |
| — | WorkflowMgmt/WorkflowController.py/WorkflowController._getVersionByWildcardPattern | ✅ | * called in getNextVersion |
| — | WorkflowMgmt/WorkflowController.py/WorkflowController._getWorkloadHandler | ✅ | * called when instantiated |
| — | WorkflowMgmt/WorkflowController.py/WorkflowController._restrictAllowedSitesByBlowUpFactor | ❌ | * called in getSiteWhiteList |
| — | WorkflowMgmt/WorkflowController.py/WorkflowController._restrictAllowedSitesByCampaign | ❌ | * called in getSiteWhiteList |
| — | WorkflowMgmt/WorkflowController.py/WorkflowController._parseOutputProcessingString | ❌ | * called in getNextVersion |
| — | WorkflowMgmt/WorkflowController.py/WorkflowController._saveInFile | ❌ | * called in getWMStats and getWMErrors |
Details
| Python 2 | Python 3 | tested | notes |
|---|---|---|---|
utils.py/workflowInfo.acquisitionEra | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getAcquisitionEraWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getAcquisitionEra | ✅ | |
utils.py/workflowInfo.campaigns | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getCampaignsWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getCampaigns | ✅ | |
utils.py/workflowInfo.getCampaigns | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getCampaigns(details=False)WorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getCampaigns(details=False) | ✅ | |
utils.py/workflowInfo._collectinchainutils.py/workflowInfo._collectin_uhm_chain | WorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getChainValues | ✅ | |
utils.py/workflowInfo.getIO | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getIOWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getIO | ✅ | |
utils.py/workflowInfo.getLumiWhiteList | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getLumiWhiteListWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getLumiWhiteList | ✅ | |
utils.py/workflowInfo.getMemory | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getMemoryWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getMemory | ✅ | |
utils.py/workflowInfo.getMulticore | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getMulticoreWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getMulticore | ✅ | |
utils.py/workflowInfo.getMulticores | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getMulticore(maxOnly=False)WorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getMulticore(maxOnly=False) | ✅ | |
utils.py/workflowInfo.processingString | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getProcessingStringWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getProcessingString | ✅ | |
utils.py/workflowInfo.getRequestNumEvents | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getRequestNumEventsWorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler.getRequestNumEvents | ✅ | |
utils.py/workflowInfo.isGoodToConvertToStepChain | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.isGoodToConvertToStepChainWorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler.isGoodToConvertToStepChain | ✅ | |
utils.py/workflowInfo.producePremix | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.isProducingPremix | ✅ | |
utils.py/workflowInfo.isRelVal | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.isRelVal | ✅ | |
| — | WorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler._getOutputSizeCorrectionFactor | ❌ | * called in checkSplitting |
| — | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler._getTaskIO | ❌ | * called in getIO |
| — | WorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler._getTimeInfo | ✅ | * called in isGoodToConvertToStepChain |
| — | WorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler._getTaskEfficiencyFactor | ❌ | * called in checkSplitting |
| — | WorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler._getTaskMaxEventsBySize | ❌ | * called in checkSplitting |
| — | WorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler._getTaskMaxEventsByTime | ❌ | * called in checkSplitting |
| — | WorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler._getTaskMaxEventsPerLumiAndJob | ❌ | * called in checkSplitting |
| — | WorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler._getTaskSchema | ❌ | * called in checkSplitting |
| — | WorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler._hasAcceptableEfficiency | ✅ | * called in isGoodToConvertToStepChain |
| — | WorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler._hasNonZeroEventStreams | ❌ | * called in isGoodToConvertToStepChain |
| — | WorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler._isBelowMinEventsPerLumi | ❌ | * called in checkSplitting |
| — | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.checkSplittingWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.checkSplittingWorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler.checkSplitting | ✅ | |
| — | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.get | ❌ | |
| — | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getBlowupFactorWorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler.getBlowupFactor | ✅ | |
| — | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getCampaignsAndLabelsWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getCampaignsAndLabels | ✅ | |
| — | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getComputingTimeWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getComputingTime | ✅ | |
| — | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getExpectedEventsPerTaskWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getExpectedEventsPerTask | ❌ | |
| — | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getParamByTaskWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getParamByTask | ❌ | |
| — | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getParamListWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getParamList | ❌ | |
| — | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getOutputDatasetsPerTaskWorkflowMgmt/WorkflowSchemaHandlers/StepChainWfSchemaHandler.py/StepChainWfSchemaHandler.getOutputDatasetsPerTask | ✅ | |
| — | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.getTasksPerOutputDatasets | ❌ | |
| — | WorkflowMgmt/WorkflowSchemaHandlers/BaseWfSchemaHandler.py/BaseWfSchemaHandler.writeDatasetPatternNameWorkflowMgmt/WorkflowSchemaHandlers/TaskChainWfSchemaHandler.py/TaskChainWfSchemaHandler.writeDatasetPatternName | ❌ |