From 3f7efd636c4bf21d5100463aa317324e55980853 Mon Sep 17 00:00:00 2001 From: Brian Toole Date: Wed, 18 May 2022 14:09:50 -0700 Subject: [PATCH 1/7] Updating instructions --- Engineer/Instructions.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/Engineer/Instructions.md b/Engineer/Instructions.md index d7e6924..04f8b42 100644 --- a/Engineer/Instructions.md +++ b/Engineer/Instructions.md @@ -7,11 +7,23 @@ For the purpose of this challenge, think of dbt as a tool that enables you to bu Data models can be configured and data tests and documentation can be added using YAML files. Click [here](https://docs.getdbt.com/docs/building-a-dbt-project/tests) for more information about dbt tests or [here](https://docs.getdbt.com/docs/building-a-dbt-project/documentation) for more information about documentation. -## Preparing for your Technical Interview +## Scenario: +A data analyst comes to the analytics engineers and says that the Care Team is struggling to meet capacity. One manager hypothesizes that this is due to an increase in the number of care projects and they need to hire more people. However, another manager thinks that there is an efficiency problem and certain tasks or projects are causing bottlenecks in the process. The data analyst needs to be able to analyze the data and deliver insights to the Care Team so they can make a decision on how to handle the capacity problem. + +## Here are some of the business questions that analyst would like to be able to answer: +How many projects are starting each week? +How many `in progress` tasks are in each project? +How long is it taking to complete tasks and projects? +Is there a particular phase of a task that takes longer than others? +Is there a particular project type or task type that takes longer than others? + +## Challenge +An analytics engineer on the team has started modeling out this data to allow the analyst to answer these questions more easily. The analytics engineer has submitted a pull request (PR) for you to review. + Prior to your technical interview, please review the files in the `base`, `intermediate`, and `prod` subdirectories as well as the open Pull Request. Make note of any questions you might have about the sample data model - you will have an opportunity to ask them during the interview. ## The Technical Interview -During your technical interview, we will discuss your review of the Pull Request in a collaborative session. You will be able to refer to any files within the model at any point for reference and you will not be asked to do any coding yourself, however we might ask you some general questions about how you would approach correcting any logic or style issues you see. +During your technical interview, we will discuss your review of the Pull Request in a collaborative session. You will not be asked to do any coding yourself, but please be prepared to share your screen and discuss what changes, suggestions, or questions you would include in your Pull Request review. Please note: You do not need to submit anything ahead of the interview or prepare a presentation of any kind. ## dbt Model structure Our dbt structure is separated across multiple databases and schemas and transformations are performed in different "layers" to reduce the repetition of logic across multiple models. From 505830c958c3026887b7c3f36a110c9746d9430c Mon Sep 17 00:00:00 2001 From: Brian Toole Date: Wed, 18 May 2022 14:12:45 -0700 Subject: [PATCH 2/7] Wording change --- Engineer/Instructions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Engineer/Instructions.md b/Engineer/Instructions.md index 04f8b42..d259053 100644 --- a/Engineer/Instructions.md +++ b/Engineer/Instructions.md @@ -20,10 +20,10 @@ Is there a particular project type or task type that takes longer than others? ## Challenge An analytics engineer on the team has started modeling out this data to allow the analyst to answer these questions more easily. The analytics engineer has submitted a pull request (PR) for you to review. -Prior to your technical interview, please review the files in the `base`, `intermediate`, and `prod` subdirectories as well as the open Pull Request. Make note of any questions you might have about the sample data model - you will have an opportunity to ask them during the interview. +Prior to your technical interview, please review the files in the `base`, `intermediate`, and `prod` subdirectories as well as the open PR. Make note of any questions you might have about the sample data model - you will have an opportunity to ask them during the interview. ## The Technical Interview -During your technical interview, we will discuss your review of the Pull Request in a collaborative session. You will not be asked to do any coding yourself, but please be prepared to share your screen and discuss what changes, suggestions, or questions you would include in your Pull Request review. Please note: You do not need to submit anything ahead of the interview or prepare a presentation of any kind. +During your technical interview, we will discuss your review of the PR in a collaborative session. You will not be asked to do any coding yourself, but please be prepared to share your screen and discuss what changes, suggestions, or questions you would include in your PR review. Please note: You do not need to submit anything ahead of the interview or prepare a presentation of any kind. ## dbt Model structure Our dbt structure is separated across multiple databases and schemas and transformations are performed in different "layers" to reduce the repetition of logic across multiple models. From ab841fc1af38f90c0462905d05ef7998c279783e Mon Sep 17 00:00:00 2001 From: Brian Toole Date: Wed, 18 May 2022 15:02:33 -0700 Subject: [PATCH 3/7] Wording correction --- Engineer/Instructions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Engineer/Instructions.md b/Engineer/Instructions.md index 04f8b42..d259053 100644 --- a/Engineer/Instructions.md +++ b/Engineer/Instructions.md @@ -20,10 +20,10 @@ Is there a particular project type or task type that takes longer than others? ## Challenge An analytics engineer on the team has started modeling out this data to allow the analyst to answer these questions more easily. The analytics engineer has submitted a pull request (PR) for you to review. -Prior to your technical interview, please review the files in the `base`, `intermediate`, and `prod` subdirectories as well as the open Pull Request. Make note of any questions you might have about the sample data model - you will have an opportunity to ask them during the interview. +Prior to your technical interview, please review the files in the `base`, `intermediate`, and `prod` subdirectories as well as the open PR. Make note of any questions you might have about the sample data model - you will have an opportunity to ask them during the interview. ## The Technical Interview -During your technical interview, we will discuss your review of the Pull Request in a collaborative session. You will not be asked to do any coding yourself, but please be prepared to share your screen and discuss what changes, suggestions, or questions you would include in your Pull Request review. Please note: You do not need to submit anything ahead of the interview or prepare a presentation of any kind. +During your technical interview, we will discuss your review of the PR in a collaborative session. You will not be asked to do any coding yourself, but please be prepared to share your screen and discuss what changes, suggestions, or questions you would include in your PR review. Please note: You do not need to submit anything ahead of the interview or prepare a presentation of any kind. ## dbt Model structure Our dbt structure is separated across multiple databases and schemas and transformations are performed in different "layers" to reduce the repetition of logic across multiple models. From 5408348948d44ffd24256ca70b5b8197f8c4daba Mon Sep 17 00:00:00 2001 From: Brian Toole Date: Wed, 18 May 2022 15:04:15 -0700 Subject: [PATCH 4/7] PR for analytics engineer interview --- Engineer/prep/base/projects.sql | 1 + Engineer/prep/intermediate/full_project_history.sql | 5 +++++ Engineer/prod/current_project_summary.sql | 12 ++++++++++-- Engineer/prod/current_task_summary.sql | 12 ++++++++++++ 4 files changed, 28 insertions(+), 2 deletions(-) create mode 100644 Engineer/prod/current_task_summary.sql diff --git a/Engineer/prep/base/projects.sql b/Engineer/prep/base/projects.sql index 2d92689..18fdd84 100644 --- a/Engineer/prep/base/projects.sql +++ b/Engineer/prep/base/projects.sql @@ -8,6 +8,7 @@ SELECT id AS project_id , modified AS modified_at , DATE(modified) AS modified_date , creator_id + , name , CASE WHEN _sdc_deleted_at IS NULL THEN FALSE ELSE TRUE END AS is_deleted_project diff --git a/Engineer/prep/intermediate/full_project_history.sql b/Engineer/prep/intermediate/full_project_history.sql index 77472f0..ce4dab7 100644 --- a/Engineer/prep/intermediate/full_project_history.sql +++ b/Engineer/prep/intermediate/full_project_history.sql @@ -1,5 +1,6 @@ SELECT COALESCE(history.project_id, projects.project_id) AS project_id , projects.creator_id -- history table does not have a project creator id field + , projects.name , COALESCE(history.project_type, projects.project_type) AS project_type , COALESCE(history.project_status, projects.project_status) AS project_status , history.previous_project_status @@ -12,6 +13,10 @@ SELECT COALESCE(history.project_id, projects.project_id) AS project_id END) OVER (PARTITION BY projects.project_id) AS first_paused_date , RANK() OVER (PARTITION BY history.project_id ORDER BY history.project_history_id DESC) AS history_sequence + , max(case when status.description = 'Complete' + then coalesce(history.modified_date, projects.modified_date) --modified_date + else null + end) over (partition by history.project_id) as last_completed_date , projects.project_deleted_date FROM {{ ref('projects') }} AS projects LEFT JOIN {{ ref('project_history') }} AS history ON history.project_id = projects.project_id \ No newline at end of file diff --git a/Engineer/prod/current_project_summary.sql b/Engineer/prod/current_project_summary.sql index e035c33..298aa82 100644 --- a/Engineer/prod/current_project_summary.sql +++ b/Engineer/prod/current_project_summary.sql @@ -1,12 +1,17 @@ SELECT full_project_history.project_id , full_project_history.creator_id + , full_project_history.name , full_project_history.project_type , full_project_history.project_status , full_project_history.previous_project_status , full_project_history.created_date , full_project_history.first_active_date + , DATEADD(DAY, -1, DATE_TRUNC(WEEK, DATE(full_project_history.first_active_date))) first_active_week , full_project_history.first_paused_date - , COUNT(full_task_history.task_id) AS number_of_tasks + , DATEADD(DAY, -1, DATE_TRUNC(WEEK, DATE(full_project_history.first_paused_date))) first_paused_week + , full_project_history.last_completed_date + , COUNT(full_task_history.task_id) AS number_of_tasks, + count(case when full_task_history.task_status = 'IN_PROGRESS' then 1 end) num_in_progress_tasks FROM {{ ref('full_project_history') }} AS full_project_history JOIN {{ ref('full_task_history') }} AS full_task_history ON full_task_history.project_id = full_project_history.project_id WHERE full_project_history.history_sequence = 1 @@ -18,4 +23,7 @@ SELECT full_project_history.project_id , full_project_history.previous_project_status , full_project_history.created_date , full_project_history.first_active_date - , full_project_history.first_paused_date \ No newline at end of file + , first_active_week + , full_project_history.first_paused_date + , first_paused_week + , full_project_history.last_completed_date \ No newline at end of file diff --git a/Engineer/prod/current_task_summary.sql b/Engineer/prod/current_task_summary.sql new file mode 100644 index 0000000..3e02f86 --- /dev/null +++ b/Engineer/prod/current_task_summary.sql @@ -0,0 +1,12 @@ +select distinct + full_task_history.task_id, + full_task_history.task_title, + full_task_history.task_type, + full_task_history.task_due_date, + full_task_history.project_id, + full_task_history.task_status, + full_task_history.first_in_progress_date, + full_task_history.first_assigned_to_adviser_date, + full_task_history.last_completed_date +from {{ ref('full_task_history') }} full_task_history +where full_task_history.history_sequence = 1 \ No newline at end of file From 6634138c462d85485cd00528dddeb048d5d5c320 Mon Sep 17 00:00:00 2001 From: Brian Toole Date: Fri, 20 May 2022 09:29:51 -0700 Subject: [PATCH 5/7] Logic simplification --- Engineer/prep/intermediate/full_project_history.sql | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Engineer/prep/intermediate/full_project_history.sql b/Engineer/prep/intermediate/full_project_history.sql index 7d9cb1d..739df5f 100644 --- a/Engineer/prep/intermediate/full_project_history.sql +++ b/Engineer/prep/intermediate/full_project_history.sql @@ -7,14 +7,14 @@ SELECT history.project_id , history.created_date , history.first_active_date , history.modified_date - , MIN(CASE WHEN status.description = 'Paused' + , MIN(CASE WHEN history.project_status = 'Paused' THEN history.modified_date ELSE NULL END) OVER (PARTITION BY history.project_id) AS first_paused_date , RANK() OVER (PARTITION BY history.project_id ORDER BY history.project_history_id DESC) AS history_sequence - , max(case when status.description = 'Complete' - then coalesce(history.modified_date, projects.modified_date) --modified_date + , max(case when history.project_status = 'Complete' + then history.modified_date else null end) over (partition by history.project_id) as last_completed_date , projects.project_deleted_date From 2cee6f5b4d5ea04a72826ebb30e7519b48ad8d98 Mon Sep 17 00:00:00 2001 From: Brian Toole Date: Fri, 20 May 2022 13:52:16 -0700 Subject: [PATCH 6/7] Adding missing name to group by --- Engineer/prod/current_project_summary.sql | 1 + 1 file changed, 1 insertion(+) diff --git a/Engineer/prod/current_project_summary.sql b/Engineer/prod/current_project_summary.sql index 298aa82..2afa326 100644 --- a/Engineer/prod/current_project_summary.sql +++ b/Engineer/prod/current_project_summary.sql @@ -18,6 +18,7 @@ SELECT full_project_history.project_id AND full_task_history.history_sequence = 1 GROUP BY full_project_history.project_id , full_project_history.creator_id + , full_project_history.name , full_project_history.project_type , full_project_history.project_status , full_project_history.previous_project_status From fba18bb288fc9ce32f550f8a01d4f9e50e3d3e85 Mon Sep 17 00:00:00 2001 From: Brian Toole Date: Thu, 26 May 2022 10:34:44 -0700 Subject: [PATCH 7/7] Updates to models and yml --- Engineer/prep/base/base.yml | 8 +++++--- Engineer/prep/intermediate/full_task_history.sql | 5 ----- Engineer/prod/current_task_summary.sql | 1 - 3 files changed, 5 insertions(+), 9 deletions(-) diff --git a/Engineer/prep/base/base.yml b/Engineer/prep/base/base.yml index 7eb4842..30c50cd 100644 --- a/Engineer/prep/base/base.yml +++ b/Engineer/prep/base/base.yml @@ -49,6 +49,10 @@ sources: description: The type of a project at the time of modification. - name: status description: The status of a project at the time of modification. + tests: + - accepted_values: + values: ['IN_PROGRESS', 'COMPLETE', 'PAUSED'] + quote: false - name: first_active description: The timestamp when a project first became active. - name: created @@ -81,9 +85,7 @@ sources: - name: project_id description: The project ID associated with a task. tests: - - not_null: - where: '_sdc_deleted_at is null' - severity: warn + - not_null - relationships: to: source ('wellthy' , 'project') field: id diff --git a/Engineer/prep/intermediate/full_task_history.sql b/Engineer/prep/intermediate/full_task_history.sql index 8b8dea5..e660a0f 100644 --- a/Engineer/prep/intermediate/full_task_history.sql +++ b/Engineer/prep/intermediate/full_task_history.sql @@ -15,11 +15,6 @@ SELECT history.task_id THEN history.modified_date ELSE NULL END) OVER (PARTITION BY history.task_id) AS first_in_progress_date - , MIN(CASE WHEN history.task_status = 'IN_PROGRESS' - AND history.adviser_id IS NOT NULL - THEN history.modified_date - ELSE NULL - END) OVER (PARTITION BY history.task_id) AS first_assigned_to_adviser_date , MAX(CASE WHEN history.task_status = 'COMPLETE' THEN history.modified_date ELSE NULL diff --git a/Engineer/prod/current_task_summary.sql b/Engineer/prod/current_task_summary.sql index 3e02f86..38a79ab 100644 --- a/Engineer/prod/current_task_summary.sql +++ b/Engineer/prod/current_task_summary.sql @@ -6,7 +6,6 @@ select distinct full_task_history.project_id, full_task_history.task_status, full_task_history.first_in_progress_date, - full_task_history.first_assigned_to_adviser_date, full_task_history.last_completed_date from {{ ref('full_task_history') }} full_task_history where full_task_history.history_sequence = 1 \ No newline at end of file