Feature/step1 by e-buerger · Pull Request #2 · workflux/exec_manager

e-buerger · 2022-05-17T12:58:46Z

setup database with sqlalchemy and sqlite
implement enums for workflow language, exec profile type and job status
implement method for creating jobs in job_factory
implement Execprofile class which only contains attributes and no methods
implement Job class which has abstract methods (prepare, exec, eval, finalize, cancel)
implement PythonJob class which inherits from Job and has the additional attribute inputs (this will be the class from which the class that is implemented by the user user inherits when he wants to use the python exec profile)
implement PyExecSession class which runs the method implemented by the user
implement tests for the functions in job_dao.py

KerstenBreuer

Vielen Dank. Ein paar generelle Hinweise. Vielleicht können wir gleich drüber gehen.

KerstenBreuer · 2022-05-19T12:04:09Z

 # limitations under the License.

-"""Short description of package""" # Please adapt to package
+"""backend""" # Please adapt to package


Suggested change
"""backend"""# Please adapt to package
"""
Apackagemanagingexecutionofjobsinawaythatisagnosticto
-theworkflowexecutionenvironment
-thelanguageusedtodescribetheworkflow
"""

KerstenBreuer · 2022-05-19T12:04:36Z

+# Copyright 2021 - 2022 Universität Tübingen, DKFZ and EMBL
+# for the German Human Genome-Phenome Archive (GHGA)


Copyright muss noch angepasst werden

KerstenBreuer · 2022-05-19T12:15:44Z

+ """enumerate workflow language types"""
+
+ CWL = "cwl"
+ """cwl language"""


Doc strings werden normalerweise nur an den Anfang von einer Klasse, Funktion, oder einem Modul gestellt.

KerstenBreuer · 2022-05-19T12:21:06Z

+from uuid import UUID
+
+from exec_manager.exec_profile import ExecProfile
+from exec_manager.job_status_type import JobStatusType
+
+
+class Job:


Job sollte am Besten eine abstrakte Klasse sein:
Suggested change
fromuuidimportUUID
fromexec_manager.exec_profileimportExecProfile
fromexec_manager.job_status_typeimportJobStatusType
classJob:
fromabcimportABC, abstractmethod
fromuuidimportUUID
fromexec_manager.exec_profileimportExecProfile
fromexec_manager.job_status_typeimportJobStatusType
classJob(ABC):

KerstenBreuer · 2022-05-19T12:23:08Z

+ def prepare(self) -> None:
+ """
+ Prepares the job.
+
+ Parameters
+ ----------
+
+ Returns
+ -------
+ NONE
+ """


Das hier sollten abstracte methoden sein:
Suggested change
defprepare(self) ->None:
"""
Preparesthejob.
Parameters
----------
Returns
-------
NONE
"""
@abstractmethod
defprepare(self) ->None:
"""
Preparesthejob.
Parameters
----------
Returns
-------
NONE
"""
...

KerstenBreuer · 2022-05-19T12:41:48Z

+
+from sqlalchemy import JSON, Column, String
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.orm.decl_api import DeclarativeMeta


Du kannst den UUID Typ von SQLalchemy benutzen:
Suggested change
fromsqlalchemy.orm.decl_apiimportDeclarativeMeta
importuuid
fromsqlalchemy.orm.decl_apiimportDeclarativeMeta
fromsqlalchemy.dialects.postgresqlimportUUID

KerstenBreuer · 2022-05-19T12:43:54Z

- id = Column(Integer, primary_key=True)
- name = Column(String, nullable=False)
- active = Column(Boolean, nullable=False)
+ job_id = Column(String, primary_key=True)


Und hier dann:
Suggested change
job_id=Column(String, primary_key=True)
job_id=Column(UUID(as_uuid=True), default=uuid.uuid4, primary_key=True)
So wird die UUID automatisch vergeben.

KerstenBreuer · 2022-05-19T12:45:06Z

+ job_id = generate_job_id()
+ job_id_str = str(job_id)


Siehe die Vorschläge im db_model. Dort habe ich einen default generator für die id eingefügt. Dann musst du dich hier nicht mehr drum kümmern.

KerstenBreuer · 2022-05-19T12:47:18Z

+from exec_manager.wf_lang_type import WfLangType
+
+
+class ExecProfile:


Für das ExecProfile bietet es sich an entweder eine Dataclass oder ein pydantic model zu verwenden, da diese Klasse lediglich daten speichern soll.
Siehe z.B. hier: https://pydantic-docs.helpmanual.io/#example

KerstenBreuer · 2022-05-19T12:50:34Z

+ exec_profile_json = json.dumps(
+ {
+ "exec_profile_type": exec_profile.exec_profile_type.value,
+ "wf_lang": exec_profile.wf_lang.value,
+ }
+ )


Wenn das ExecProfile auf einem pydantic model beruht kann man einfach schreiben:
Suggested change
exec_profile_json=json.dumps(
{
"exec_profile_type": exec_profile.exec_profile_type.value,
"wf_lang": exec_profile.wf_lang.value,
}
)
exec_profile_dict=exec_profile.to_dict()
Das hier returned ein dict. Also kein JSON. Aber die Conversion zu JSON kann man auch sqlalchemy machen lassen.

create classfiles adapting code for passing precommit adapt code for passing precommit adapt code for passing precommit implement a few functions change type of db columns satisfy pre-commit implement exec method of PythonJob suppres bandit warnigns and add comments remove __init__.py from dao directory remove context manager fix session error in job_dao.py change db commands change db commands change db commands fix import errors by using absolute path and an __init__ file remove content from exec method add license headers add license headers implement PyExecSession implement unit tests for job_dao satisfy pipeline satisfy pipeline add license header update README update doc strings; change job_dao functions for better unit tests change engine type

e-buerger added 27 commits May 4, 2022 09:07

create classfiles
fd7fce9

adapting code for passing precommit
dad4372

adapt code for passing precommit
f59a516

adapt code for passing precommit
a19747c

implement a few functions
7588dde

change type of db columns
56af3aa

satisfy pre-commit
566dbe1

implement exec method of PythonJob
37ae4a4

suppres bandit warnigns and add comments
5529827

remove __init__.py from dao directory
5f17ab3

remove context manager
cd6f426

fix session error in job_dao.py
792ce11

change db commands
4eeafcf

change db commands
af7d7a9

change db commands
a1d9e61

fix import errors by using absolute path and an __init__ file
1993d54

remove content from exec method
ae53782

add license headers
211f348

add license headers
46f313e

implement PyExecSession
c5253c3

implement unit tests for job_dao
97c902a

satisfy pipeline
6491949

satisfy pipeline
c344999

add license header
d93859c

update README
c862e18

update doc strings; change job_dao functions for better unit tests
37b711c

change engine type
bf7255d

e-buerger requested a review from KerstenBreuer May 17, 2022 12:58

KerstenBreuer requested changes May 19, 2022
View reviewed changes

KerstenBreuer force-pushed the feature/step1 branch from bf7255d to f671e43Compare May 19, 2022 13:23

KerstenBreuerand others added 6 commits May 19, 2022 14:15

refactor
c286f37

resolve merge conflicts
5ab918d

restructure modules
9370930

remove unneccassery class
e5e559b

satisfy pipeline
86b72dd

update doc strings
9856610

Navigation Menu

Uh oh!

Feature/step1 - #2

Feature/step1#2
e-buerger wants to merge 34 commits into
mainfrom
feature/step1

e-buerger commented May 17, 2022

Uh oh!

KerstenBreuer left a comment

Uh oh!

KerstenBreuerMay 19, 2022

Uh oh!

KerstenBreuerMay 19, 2022

Uh oh!

KerstenBreuerMay 19, 2022

Uh oh!

KerstenBreuerMay 19, 2022

Uh oh!

KerstenBreuerMay 19, 2022

Uh oh!

KerstenBreuerMay 19, 2022

Uh oh!

KerstenBreuerMay 19, 2022

Uh oh!

KerstenBreuerMay 19, 2022

Uh oh!

KerstenBreuerMay 19, 2022

Uh oh!

KerstenBreuerMay 19, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-"""backend"""# Please adapt to package
+"""
+Apackagemanagingexecutionofjobsinawaythatisagnosticto
+-theworkflowexecutionenvironment
+-thelanguageusedtodescribetheworkflow
+"""

		# Copyright 2021 - 2022 Universität Tübingen, DKFZ and EMBL
		# for the German Human Genome-Phenome Archive (GHGA)

-fromuuidimportUUID
-fromexec_manager.exec_profileimportExecProfile
-fromexec_manager.job_status_typeimportJobStatusType
-classJob:
+fromabcimportABC, abstractmethod
+fromuuidimportUUID
+fromexec_manager.exec_profileimportExecProfile
+fromexec_manager.job_status_typeimportJobStatusType
+classJob(ABC):

-defprepare(self) ->None:
- """
-Preparesthejob.
-Parameters
-----------
-Returns
--------
-NONE
- """
+@abstractmethod
+defprepare(self) ->None:
+ """
+Preparesthejob.
+Parameters
+----------
+Returns
+-------
+NONE
+ """
+ ...

-fromsqlalchemy.orm.decl_apiimportDeclarativeMeta
+importuuid
+fromsqlalchemy.orm.decl_apiimportDeclarativeMeta
+fromsqlalchemy.dialects.postgresqlimportUUID

	job_id=Column(String, primary_key=True)
	job_id=Column(UUID(as_uuid=True), default=uuid.uuid4, primary_key=True)

		from exec_manager.wf_lang_type import WfLangType


		class ExecProfile:

Uh oh!

Conversation

e-buerger commented May 17, 2022

Uh oh!

KerstenBreuer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants