FOUR-20944: Implement ETag Caching for Task and Case Data by eiresendez · Pull Request #7786 · ProcessMaker/processmaker

eiresendez · 2024-12-02T18:23:16Z

Issue & Reproduction Steps

This PR implements ETag caching for all GET and HEAD requests, enhancing performance and reducing bandwidth usage by leveraging HTTP conditional requests. The middleware is now applied globally, ensuring consistent ETag handling across endpoints.

Solution

Global ETag Middleware:

Applied the ETag middleware to all GET API requests to ensure consistency and avoid redundant implementation for individual routes.
Middleware calculates and validates ETags based on the response or a configurable sources of truth (optional).

Exclude active_at Field:

Removed the active_at field from TaskResource and requestor serialization to prevent ETag inconsistencies.

Introduce Configurable ETag Source:

updated_at: Default method using database timestamps to determine the latest update for relevant tables.

How to Test

Call any GET endpoint for the first time. A 200 OK response is returned and the content is fetched from the server and cached.
Refresh the page or call the endpoint again. A 304 Not Modified response is returned if the ETag matches.
Modify the relevant data (e.g., update the database). On subsequent requests:
• If using updated_at as the source, a 200 OK response is returned with the updated content.
Validate that the updated content is cached and subsequent requests return 304 Not Modified.

Related Tickets & Packages

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

estebangallego

Works as expected! 🔥

processmaker-sonarqube · 2024-12-06T16:24:20Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

eiresendez · 2024-12-02T18:23:16Z

Issue & Reproduction Steps

This PR implements ETag caching for all GET and HEAD requests, enhancing performance and reducing bandwidth usage by leveraging HTTP conditional requests. The middleware is now applied globally, ensuring consistent ETag handling across endpoints.

Solution

Global ETag Middleware:

Applied the ETag middleware to all GET API requests to ensure consistency and avoid redundant implementation for individual routes.
Middleware calculates and validates ETags based on the response or a configurable sources of truth (optional).

Exclude active_at Field:

Removed the active_at field from TaskResource and requestor serialization to prevent ETag inconsistencies.

Introduce Configurable ETag Source:

updated_at: Default method using database timestamps to determine the latest update for relevant tables.

How to Test

Call any GET endpoint for the first time. A 200 OK response is returned and the content is fetched from the server and cached.
Refresh the page or call the endpoint again. A 304 Not Modified response is returned if the ETag matches.
Modify the relevant data (e.g., update the database). On subsequent requests:
• If using updated_at as the source, a 200 OK response is returned with the updated content.
Validate that the updated content is cached and subsequent requests return 304 Not Modified.

Related Tickets & Packages

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

estebangallego

Works as expected! 🔥

processmaker-sonarqube · 2024-12-06T16:24:20Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

eiresendez · 2024-12-02T18:23:16Z

Issue & Reproduction Steps

This PR implements ETag caching for all GET and HEAD requests, enhancing performance and reducing bandwidth usage by leveraging HTTP conditional requests. The middleware is now applied globally, ensuring consistent ETag handling across endpoints.

Solution

Global ETag Middleware:

Applied the ETag middleware to all GET API requests to ensure consistency and avoid redundant implementation for individual routes.
Middleware calculates and validates ETags based on the response or a configurable sources of truth (optional).

Exclude active_at Field:

Removed the active_at field from TaskResource and requestor serialization to prevent ETag inconsistencies.

Introduce Configurable ETag Source:

updated_at: Default method using database timestamps to determine the latest update for relevant tables.

How to Test

Call any GET endpoint for the first time. A 200 OK response is returned and the content is fetched from the server and cached.
Refresh the page or call the endpoint again. A 304 Not Modified response is returned if the ETag matches.
Modify the relevant data (e.g., update the database). On subsequent requests:
• If using updated_at as the source, a 200 OK response is returned with the updated content.
Validate that the updated content is cached and subsequent requests return 304 Not Modified.

Related Tickets & Packages

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

estebangallego

Works as expected! 🔥

processmaker-sonarqube · 2024-12-06T16:24:20Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

eiresendez · 2024-12-02T18:23:16Z

Issue & Reproduction Steps

This PR implements ETag caching for all GET and HEAD requests, enhancing performance and reducing bandwidth usage by leveraging HTTP conditional requests. The middleware is now applied globally, ensuring consistent ETag handling across endpoints.

Solution

Global ETag Middleware:

Applied the ETag middleware to all GET API requests to ensure consistency and avoid redundant implementation for individual routes.
Middleware calculates and validates ETags based on the response or a configurable sources of truth (optional).

Exclude active_at Field:

Removed the active_at field from TaskResource and requestor serialization to prevent ETag inconsistencies.

Introduce Configurable ETag Source:

updated_at: Default method using database timestamps to determine the latest update for relevant tables.

How to Test

Call any GET endpoint for the first time. A 200 OK response is returned and the content is fetched from the server and cached.
Refresh the page or call the endpoint again. A 304 Not Modified response is returned if the ETag matches.
Modify the relevant data (e.g., update the database). On subsequent requests:
• If using updated_at as the source, a 200 OK response is returned with the updated content.
Validate that the updated content is cached and subsequent requests return 304 Not Modified.

Related Tickets & Packages

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

estebangallego

Works as expected! 🔥

processmaker-sonarqube · 2024-12-06T16:24:20Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

eiresendez · 2024-12-02T18:23:16Z

Issue & Reproduction Steps

This PR implements ETag caching for all GET and HEAD requests, enhancing performance and reducing bandwidth usage by leveraging HTTP conditional requests. The middleware is now applied globally, ensuring consistent ETag handling across endpoints.

Solution

Global ETag Middleware:

Applied the ETag middleware to all GET API requests to ensure consistency and avoid redundant implementation for individual routes.
Middleware calculates and validates ETags based on the response or a configurable sources of truth (optional).

Exclude active_at Field:

Removed the active_at field from TaskResource and requestor serialization to prevent ETag inconsistencies.

Introduce Configurable ETag Source:

updated_at: Default method using database timestamps to determine the latest update for relevant tables.

How to Test

Call any GET endpoint for the first time. A 200 OK response is returned and the content is fetched from the server and cached.
Refresh the page or call the endpoint again. A 304 Not Modified response is returned if the ETag matches.
Modify the relevant data (e.g., update the database). On subsequent requests:
• If using updated_at as the source, a 200 OK response is returned with the updated content.
Validate that the updated content is cached and subsequent requests return 304 Not Modified.

Related Tickets & Packages

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

estebangallego

Works as expected! 🔥

processmaker-sonarqube · 2024-12-06T16:24:20Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

eiresendez · 2024-12-02T18:23:16Z

Issue & Reproduction Steps

This PR implements ETag caching for all GET and HEAD requests, enhancing performance and reducing bandwidth usage by leveraging HTTP conditional requests. The middleware is now applied globally, ensuring consistent ETag handling across endpoints.

Solution

Global ETag Middleware:

Applied the ETag middleware to all GET API requests to ensure consistency and avoid redundant implementation for individual routes.
Middleware calculates and validates ETags based on the response or a configurable sources of truth (optional).

Exclude active_at Field:

Removed the active_at field from TaskResource and requestor serialization to prevent ETag inconsistencies.

Introduce Configurable ETag Source:

updated_at: Default method using database timestamps to determine the latest update for relevant tables.

How to Test

Call any GET endpoint for the first time. A 200 OK response is returned and the content is fetched from the server and cached.
Refresh the page or call the endpoint again. A 304 Not Modified response is returned if the ETag matches.
Modify the relevant data (e.g., update the database). On subsequent requests:
• If using updated_at as the source, a 200 OK response is returned with the updated content.
Validate that the updated content is cached and subsequent requests return 304 Not Modified.

Related Tickets & Packages

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

estebangallego

Works as expected! 🔥

processmaker-sonarqube · 2024-12-06T16:24:20Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

eiresendez · 2024-12-02T18:23:16Z

Issue & Reproduction Steps

This PR implements ETag caching for all GET and HEAD requests, enhancing performance and reducing bandwidth usage by leveraging HTTP conditional requests. The middleware is now applied globally, ensuring consistent ETag handling across endpoints.

Solution

Global ETag Middleware:

Applied the ETag middleware to all GET API requests to ensure consistency and avoid redundant implementation for individual routes.
Middleware calculates and validates ETags based on the response or a configurable sources of truth (optional).

Exclude active_at Field:

Removed the active_at field from TaskResource and requestor serialization to prevent ETag inconsistencies.

Introduce Configurable ETag Source:

updated_at: Default method using database timestamps to determine the latest update for relevant tables.

How to Test

Call any GET endpoint for the first time. A 200 OK response is returned and the content is fetched from the server and cached.
Refresh the page or call the endpoint again. A 304 Not Modified response is returned if the ETag matches.
Modify the relevant data (e.g., update the database). On subsequent requests:
• If using updated_at as the source, a 200 OK response is returned with the updated content.
Validate that the updated content is cached and subsequent requests return 304 Not Modified.

Related Tickets & Packages

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

estebangallego

Works as expected! 🔥

processmaker-sonarqube · 2024-12-06T16:24:20Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

eiresendez · 2024-12-02T18:23:16Z

Issue & Reproduction Steps

This PR implements ETag caching for all GET and HEAD requests, enhancing performance and reducing bandwidth usage by leveraging HTTP conditional requests. The middleware is now applied globally, ensuring consistent ETag handling across endpoints.

Solution

Global ETag Middleware:

Applied the ETag middleware to all GET API requests to ensure consistency and avoid redundant implementation for individual routes.
Middleware calculates and validates ETags based on the response or a configurable sources of truth (optional).

Exclude active_at Field:

Removed the active_at field from TaskResource and requestor serialization to prevent ETag inconsistencies.

Introduce Configurable ETag Source:

updated_at: Default method using database timestamps to determine the latest update for relevant tables.

How to Test

Call any GET endpoint for the first time. A 200 OK response is returned and the content is fetched from the server and cached.
Refresh the page or call the endpoint again. A 304 Not Modified response is returned if the ETag matches.
Modify the relevant data (e.g., update the database). On subsequent requests:
• If using updated_at as the source, a 200 OK response is returned with the updated content.
Validate that the updated content is cached and subsequent requests return 304 Not Modified.

Related Tickets & Packages

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

estebangallego

Works as expected! 🔥

processmaker-sonarqube · 2024-12-06T16:24:20Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

eiresendez added 6 commits December 2, 2024 11:05

Exclude 'active_at' field from TaskResource and requestor serializati…
f058177
…on to prevent ETag inconsistencies

Implement ETag Caching for Task Responses
48cd7d6

Add ETag middleware to API group to support package routes
dae2468

Apply ETag middleware to v1.1 routes for consistent caching
8a94369

eiresendez requested a review from estebangallego December 5, 2024 22:05

estebangallego reviewed Dec 6, 2024
View reviewed changes

Comment threadProcessMaker/Providers/RouteServiceProvider.php

add ETag middleware to startProcesses route
cdaa80d
- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

eiresendez requested a review from estebangallego December 6, 2024 15:44

estebangallego approved these changes Dec 6, 2024
View reviewed changes

eiresendez merged commit e64f7a7 into epic/FOUR-20929Dec 6, 2024