FOUR-20929: Implement ETag Caching by eiresendez · Pull Request #7892 · ProcessMaker/processmaker

eiresendez · 2025-01-13T17:51:34Z

Issue & Reproduction Steps

The server sends large amounts of data with each request, even if the data hasn’t changed. This causes unnecessary server load, increased response times and higher resource consumption, especially for repeated requests.

Solution

Added ETag caching headers to API endpoints.
The server will now respond with a 304 Not Modified when data remains unchanged, improving performance and reducing bandwidth usage.

How to Test

Verify that repeated requests for the same unchanged data return a 304 Not Modified status.
Check the docs/etag-caching.md for detailed steps and testing guidelines.

Related Tickets & Packages

FOUR-20929

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

- Introduced IfNoneMatch middleware to handle ETag-based caching: - Supports safe HTTP methods (GET, HEAD). - Returns "304 Not Modified" for matching ETags. - Handles multiple ETags and restores original request methods. - Added EtagManager for flexible ETag generation: - Default ETag generation uses MD5 hash. - Supports custom hash algorithms like SHA-256 via callbacks. - Included test cases: - Validate default ETag generation. - Test custom callback logic for ETag creation. This commit adds foundational support for ETag-based caching in the application.

…ware - Merged `etag.set` and `etag.if-none-match` middlewares into a single `etag.handle` middleware. - Simplified logic to reduce redundancy and improve maintainability. - Ensured ETag validation (`If-None-Match`) and generation are handled in the same flow. - Preserved compatibility with HEAD requests for consistency. This refactor improves clarity, reduces potential misconfigurations, and keeps the ETag logic centralized.

…d tests - Created tests for the new `HandleEtag` middleware: - Validates ETag generation and correctness. - Tests responses for both matching and non-matching `If-None-Match` headers. - Ensures proper handling of weak ETags (`W/`). - Removed old tests for `SetEtag` and `IfNoneMatch` middlewares as they are no longer needed. This commit improves test clarity and ensures the new ETag middleware behaves as expected.

- Added a test to validate ETag generation for user-specific routes using `etag:user`. - Simulates an authenticated user and verifies the ETag includes the user ID. These tests ensure the ETag middleware behaves correctly for both user-specific and common routes.

FOUR-20933: Implement ETag Caching for Screens Data

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

FOUR-20944: Implement ETag Caching for Task and Case Data

- Includes an overview of ETag functionality and benefits. - Provides implementation details for the HandleEtag middleware. - Covers ETag generation based on tables and response content. - Adds examples for conditional responses and route configuration. - Documents unit and manual testing approaches for ETag functionality. - Suggests future improvements for ETag caching and metrics collection.

- Introduced a new test class `HandleEtagCacheInvalidationTest` to verify ETag behavior upon database updates. - Implemented tests to ensure ETag changes when the underlying data is modified and that the correct ETag is returned for subsequent requests. - Updated existing `HandleEtagTest` to include a test for returning 304 Not Modified when the ETag matches the client-provided value.

- Introduced logging for highly dynamic endpoints by tracking ETag history for each endpoint. - Implemented a mechanism to limit the number of tracked ETags and log when all tracked ETags are unique. - Updated documentation to reflect the new logging feature and its implications for performance optimization.

- Integrated `config/etag.php` for dynamic configuration of ETag functionality. - Added `enabled` and `log_dynamic_endpoints` flags to control feature behavior. This update improves flexibility and allows disabling ETag processing entirely when `enabled` is set to false.

- Added custom Trend metrics to measure and compare durations of 200 OK and 304 Not Modified responses - Validates that 304 responses are faster using If-None-Match header - Improved test clarity by focusing on ETag performance under load

- Implemented tracking of ETag values for specified endpoints - Added logic to identify dynamic endpoints when ETag history shows diff values

FOUR-20954: Monitor ETag Caching Performance

processmaker-sonarqube · 2025-01-23T16:16:36Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

eiresendez · 2025-01-13T17:51:34Z

Issue & Reproduction Steps

The server sends large amounts of data with each request, even if the data hasn’t changed. This causes unnecessary server load, increased response times and higher resource consumption, especially for repeated requests.

Solution

Added ETag caching headers to API endpoints.
The server will now respond with a 304 Not Modified when data remains unchanged, improving performance and reducing bandwidth usage.

How to Test

Verify that repeated requests for the same unchanged data return a 304 Not Modified status.
Check the docs/etag-caching.md for detailed steps and testing guidelines.

Related Tickets & Packages

FOUR-20929

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

- Introduced IfNoneMatch middleware to handle ETag-based caching: - Supports safe HTTP methods (GET, HEAD). - Returns "304 Not Modified" for matching ETags. - Handles multiple ETags and restores original request methods. - Added EtagManager for flexible ETag generation: - Default ETag generation uses MD5 hash. - Supports custom hash algorithms like SHA-256 via callbacks. - Included test cases: - Validate default ETag generation. - Test custom callback logic for ETag creation. This commit adds foundational support for ETag-based caching in the application.

…ware - Merged `etag.set` and `etag.if-none-match` middlewares into a single `etag.handle` middleware. - Simplified logic to reduce redundancy and improve maintainability. - Ensured ETag validation (`If-None-Match`) and generation are handled in the same flow. - Preserved compatibility with HEAD requests for consistency. This refactor improves clarity, reduces potential misconfigurations, and keeps the ETag logic centralized.

…d tests - Created tests for the new `HandleEtag` middleware: - Validates ETag generation and correctness. - Tests responses for both matching and non-matching `If-None-Match` headers. - Ensures proper handling of weak ETags (`W/`). - Removed old tests for `SetEtag` and `IfNoneMatch` middlewares as they are no longer needed. This commit improves test clarity and ensures the new ETag middleware behaves as expected.

- Added a test to validate ETag generation for user-specific routes using `etag:user`. - Simulates an authenticated user and verifies the ETag includes the user ID. These tests ensure the ETag middleware behaves correctly for both user-specific and common routes.

FOUR-20933: Implement ETag Caching for Screens Data

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

FOUR-20944: Implement ETag Caching for Task and Case Data

- Includes an overview of ETag functionality and benefits. - Provides implementation details for the HandleEtag middleware. - Covers ETag generation based on tables and response content. - Adds examples for conditional responses and route configuration. - Documents unit and manual testing approaches for ETag functionality. - Suggests future improvements for ETag caching and metrics collection.

- Introduced a new test class `HandleEtagCacheInvalidationTest` to verify ETag behavior upon database updates. - Implemented tests to ensure ETag changes when the underlying data is modified and that the correct ETag is returned for subsequent requests. - Updated existing `HandleEtagTest` to include a test for returning 304 Not Modified when the ETag matches the client-provided value.

- Introduced logging for highly dynamic endpoints by tracking ETag history for each endpoint. - Implemented a mechanism to limit the number of tracked ETags and log when all tracked ETags are unique. - Updated documentation to reflect the new logging feature and its implications for performance optimization.

- Integrated `config/etag.php` for dynamic configuration of ETag functionality. - Added `enabled` and `log_dynamic_endpoints` flags to control feature behavior. This update improves flexibility and allows disabling ETag processing entirely when `enabled` is set to false.

- Added custom Trend metrics to measure and compare durations of 200 OK and 304 Not Modified responses - Validates that 304 responses are faster using If-None-Match header - Improved test clarity by focusing on ETag performance under load

- Implemented tracking of ETag values for specified endpoints - Added logic to identify dynamic endpoints when ETag history shows diff values

FOUR-20954: Monitor ETag Caching Performance

processmaker-sonarqube · 2025-01-23T16:16:36Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

eiresendez · 2025-01-13T17:51:34Z

Issue & Reproduction Steps

The server sends large amounts of data with each request, even if the data hasn’t changed. This causes unnecessary server load, increased response times and higher resource consumption, especially for repeated requests.

Solution

Added ETag caching headers to API endpoints.
The server will now respond with a 304 Not Modified when data remains unchanged, improving performance and reducing bandwidth usage.

How to Test

Verify that repeated requests for the same unchanged data return a 304 Not Modified status.
Check the docs/etag-caching.md for detailed steps and testing guidelines.

Related Tickets & Packages

FOUR-20929

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

- Introduced IfNoneMatch middleware to handle ETag-based caching: - Supports safe HTTP methods (GET, HEAD). - Returns "304 Not Modified" for matching ETags. - Handles multiple ETags and restores original request methods. - Added EtagManager for flexible ETag generation: - Default ETag generation uses MD5 hash. - Supports custom hash algorithms like SHA-256 via callbacks. - Included test cases: - Validate default ETag generation. - Test custom callback logic for ETag creation. This commit adds foundational support for ETag-based caching in the application.

…ware - Merged `etag.set` and `etag.if-none-match` middlewares into a single `etag.handle` middleware. - Simplified logic to reduce redundancy and improve maintainability. - Ensured ETag validation (`If-None-Match`) and generation are handled in the same flow. - Preserved compatibility with HEAD requests for consistency. This refactor improves clarity, reduces potential misconfigurations, and keeps the ETag logic centralized.

…d tests - Created tests for the new `HandleEtag` middleware: - Validates ETag generation and correctness. - Tests responses for both matching and non-matching `If-None-Match` headers. - Ensures proper handling of weak ETags (`W/`). - Removed old tests for `SetEtag` and `IfNoneMatch` middlewares as they are no longer needed. This commit improves test clarity and ensures the new ETag middleware behaves as expected.

- Added a test to validate ETag generation for user-specific routes using `etag:user`. - Simulates an authenticated user and verifies the ETag includes the user ID. These tests ensure the ETag middleware behaves correctly for both user-specific and common routes.

FOUR-20933: Implement ETag Caching for Screens Data

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

FOUR-20944: Implement ETag Caching for Task and Case Data

- Includes an overview of ETag functionality and benefits. - Provides implementation details for the HandleEtag middleware. - Covers ETag generation based on tables and response content. - Adds examples for conditional responses and route configuration. - Documents unit and manual testing approaches for ETag functionality. - Suggests future improvements for ETag caching and metrics collection.

- Introduced a new test class `HandleEtagCacheInvalidationTest` to verify ETag behavior upon database updates. - Implemented tests to ensure ETag changes when the underlying data is modified and that the correct ETag is returned for subsequent requests. - Updated existing `HandleEtagTest` to include a test for returning 304 Not Modified when the ETag matches the client-provided value.

- Introduced logging for highly dynamic endpoints by tracking ETag history for each endpoint. - Implemented a mechanism to limit the number of tracked ETags and log when all tracked ETags are unique. - Updated documentation to reflect the new logging feature and its implications for performance optimization.

- Integrated `config/etag.php` for dynamic configuration of ETag functionality. - Added `enabled` and `log_dynamic_endpoints` flags to control feature behavior. This update improves flexibility and allows disabling ETag processing entirely when `enabled` is set to false.

- Added custom Trend metrics to measure and compare durations of 200 OK and 304 Not Modified responses - Validates that 304 responses are faster using If-None-Match header - Improved test clarity by focusing on ETag performance under load

- Implemented tracking of ETag values for specified endpoints - Added logic to identify dynamic endpoints when ETag history shows diff values

FOUR-20954: Monitor ETag Caching Performance

processmaker-sonarqube · 2025-01-23T16:16:36Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

eiresendez · 2025-01-13T17:51:34Z

Issue & Reproduction Steps

The server sends large amounts of data with each request, even if the data hasn’t changed. This causes unnecessary server load, increased response times and higher resource consumption, especially for repeated requests.

Solution

Added ETag caching headers to API endpoints.
The server will now respond with a 304 Not Modified when data remains unchanged, improving performance and reducing bandwidth usage.

How to Test

Verify that repeated requests for the same unchanged data return a 304 Not Modified status.
Check the docs/etag-caching.md for detailed steps and testing guidelines.

Related Tickets & Packages

FOUR-20929

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

- Introduced IfNoneMatch middleware to handle ETag-based caching: - Supports safe HTTP methods (GET, HEAD). - Returns "304 Not Modified" for matching ETags. - Handles multiple ETags and restores original request methods. - Added EtagManager for flexible ETag generation: - Default ETag generation uses MD5 hash. - Supports custom hash algorithms like SHA-256 via callbacks. - Included test cases: - Validate default ETag generation. - Test custom callback logic for ETag creation. This commit adds foundational support for ETag-based caching in the application.

…ware - Merged `etag.set` and `etag.if-none-match` middlewares into a single `etag.handle` middleware. - Simplified logic to reduce redundancy and improve maintainability. - Ensured ETag validation (`If-None-Match`) and generation are handled in the same flow. - Preserved compatibility with HEAD requests for consistency. This refactor improves clarity, reduces potential misconfigurations, and keeps the ETag logic centralized.

…d tests - Created tests for the new `HandleEtag` middleware: - Validates ETag generation and correctness. - Tests responses for both matching and non-matching `If-None-Match` headers. - Ensures proper handling of weak ETags (`W/`). - Removed old tests for `SetEtag` and `IfNoneMatch` middlewares as they are no longer needed. This commit improves test clarity and ensures the new ETag middleware behaves as expected.

- Added a test to validate ETag generation for user-specific routes using `etag:user`. - Simulates an authenticated user and verifies the ETag includes the user ID. These tests ensure the ETag middleware behaves correctly for both user-specific and common routes.

FOUR-20933: Implement ETag Caching for Screens Data

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

FOUR-20944: Implement ETag Caching for Task and Case Data

- Includes an overview of ETag functionality and benefits. - Provides implementation details for the HandleEtag middleware. - Covers ETag generation based on tables and response content. - Adds examples for conditional responses and route configuration. - Documents unit and manual testing approaches for ETag functionality. - Suggests future improvements for ETag caching and metrics collection.

- Introduced a new test class `HandleEtagCacheInvalidationTest` to verify ETag behavior upon database updates. - Implemented tests to ensure ETag changes when the underlying data is modified and that the correct ETag is returned for subsequent requests. - Updated existing `HandleEtagTest` to include a test for returning 304 Not Modified when the ETag matches the client-provided value.

- Introduced logging for highly dynamic endpoints by tracking ETag history for each endpoint. - Implemented a mechanism to limit the number of tracked ETags and log when all tracked ETags are unique. - Updated documentation to reflect the new logging feature and its implications for performance optimization.

- Integrated `config/etag.php` for dynamic configuration of ETag functionality. - Added `enabled` and `log_dynamic_endpoints` flags to control feature behavior. This update improves flexibility and allows disabling ETag processing entirely when `enabled` is set to false.

- Added custom Trend metrics to measure and compare durations of 200 OK and 304 Not Modified responses - Validates that 304 responses are faster using If-None-Match header - Improved test clarity by focusing on ETag performance under load

- Implemented tracking of ETag values for specified endpoints - Added logic to identify dynamic endpoints when ETag history shows diff values

FOUR-20954: Monitor ETag Caching Performance

processmaker-sonarqube · 2025-01-23T16:16:36Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

eiresendez · 2025-01-13T17:51:34Z

Issue & Reproduction Steps

The server sends large amounts of data with each request, even if the data hasn’t changed. This causes unnecessary server load, increased response times and higher resource consumption, especially for repeated requests.

Solution

Added ETag caching headers to API endpoints.
The server will now respond with a 304 Not Modified when data remains unchanged, improving performance and reducing bandwidth usage.

How to Test

Verify that repeated requests for the same unchanged data return a 304 Not Modified status.
Check the docs/etag-caching.md for detailed steps and testing guidelines.

Related Tickets & Packages

FOUR-20929

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

- Introduced IfNoneMatch middleware to handle ETag-based caching: - Supports safe HTTP methods (GET, HEAD). - Returns "304 Not Modified" for matching ETags. - Handles multiple ETags and restores original request methods. - Added EtagManager for flexible ETag generation: - Default ETag generation uses MD5 hash. - Supports custom hash algorithms like SHA-256 via callbacks. - Included test cases: - Validate default ETag generation. - Test custom callback logic for ETag creation. This commit adds foundational support for ETag-based caching in the application.

…ware - Merged `etag.set` and `etag.if-none-match` middlewares into a single `etag.handle` middleware. - Simplified logic to reduce redundancy and improve maintainability. - Ensured ETag validation (`If-None-Match`) and generation are handled in the same flow. - Preserved compatibility with HEAD requests for consistency. This refactor improves clarity, reduces potential misconfigurations, and keeps the ETag logic centralized.

…d tests - Created tests for the new `HandleEtag` middleware: - Validates ETag generation and correctness. - Tests responses for both matching and non-matching `If-None-Match` headers. - Ensures proper handling of weak ETags (`W/`). - Removed old tests for `SetEtag` and `IfNoneMatch` middlewares as they are no longer needed. This commit improves test clarity and ensures the new ETag middleware behaves as expected.

- Added a test to validate ETag generation for user-specific routes using `etag:user`. - Simulates an authenticated user and verifies the ETag includes the user ID. These tests ensure the ETag middleware behaves correctly for both user-specific and common routes.

FOUR-20933: Implement ETag Caching for Screens Data

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

FOUR-20944: Implement ETag Caching for Task and Case Data

- Includes an overview of ETag functionality and benefits. - Provides implementation details for the HandleEtag middleware. - Covers ETag generation based on tables and response content. - Adds examples for conditional responses and route configuration. - Documents unit and manual testing approaches for ETag functionality. - Suggests future improvements for ETag caching and metrics collection.

- Introduced a new test class `HandleEtagCacheInvalidationTest` to verify ETag behavior upon database updates. - Implemented tests to ensure ETag changes when the underlying data is modified and that the correct ETag is returned for subsequent requests. - Updated existing `HandleEtagTest` to include a test for returning 304 Not Modified when the ETag matches the client-provided value.

- Introduced logging for highly dynamic endpoints by tracking ETag history for each endpoint. - Implemented a mechanism to limit the number of tracked ETags and log when all tracked ETags are unique. - Updated documentation to reflect the new logging feature and its implications for performance optimization.

- Integrated `config/etag.php` for dynamic configuration of ETag functionality. - Added `enabled` and `log_dynamic_endpoints` flags to control feature behavior. This update improves flexibility and allows disabling ETag processing entirely when `enabled` is set to false.

- Added custom Trend metrics to measure and compare durations of 200 OK and 304 Not Modified responses - Validates that 304 responses are faster using If-None-Match header - Improved test clarity by focusing on ETag performance under load

- Implemented tracking of ETag values for specified endpoints - Added logic to identify dynamic endpoints when ETag history shows diff values

FOUR-20954: Monitor ETag Caching Performance

processmaker-sonarqube · 2025-01-23T16:16:36Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

eiresendez · 2025-01-13T17:51:34Z

Issue & Reproduction Steps

The server sends large amounts of data with each request, even if the data hasn’t changed. This causes unnecessary server load, increased response times and higher resource consumption, especially for repeated requests.

Solution

Added ETag caching headers to API endpoints.
The server will now respond with a 304 Not Modified when data remains unchanged, improving performance and reducing bandwidth usage.

How to Test

Verify that repeated requests for the same unchanged data return a 304 Not Modified status.
Check the docs/etag-caching.md for detailed steps and testing guidelines.

Related Tickets & Packages

FOUR-20929

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

- Introduced IfNoneMatch middleware to handle ETag-based caching: - Supports safe HTTP methods (GET, HEAD). - Returns "304 Not Modified" for matching ETags. - Handles multiple ETags and restores original request methods. - Added EtagManager for flexible ETag generation: - Default ETag generation uses MD5 hash. - Supports custom hash algorithms like SHA-256 via callbacks. - Included test cases: - Validate default ETag generation. - Test custom callback logic for ETag creation. This commit adds foundational support for ETag-based caching in the application.

…ware - Merged `etag.set` and `etag.if-none-match` middlewares into a single `etag.handle` middleware. - Simplified logic to reduce redundancy and improve maintainability. - Ensured ETag validation (`If-None-Match`) and generation are handled in the same flow. - Preserved compatibility with HEAD requests for consistency. This refactor improves clarity, reduces potential misconfigurations, and keeps the ETag logic centralized.

…d tests - Created tests for the new `HandleEtag` middleware: - Validates ETag generation and correctness. - Tests responses for both matching and non-matching `If-None-Match` headers. - Ensures proper handling of weak ETags (`W/`). - Removed old tests for `SetEtag` and `IfNoneMatch` middlewares as they are no longer needed. This commit improves test clarity and ensures the new ETag middleware behaves as expected.

- Added a test to validate ETag generation for user-specific routes using `etag:user`. - Simulates an authenticated user and verifies the ETag includes the user ID. These tests ensure the ETag middleware behaves correctly for both user-specific and common routes.

FOUR-20933: Implement ETag Caching for Screens Data

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

FOUR-20944: Implement ETag Caching for Task and Case Data

- Includes an overview of ETag functionality and benefits. - Provides implementation details for the HandleEtag middleware. - Covers ETag generation based on tables and response content. - Adds examples for conditional responses and route configuration. - Documents unit and manual testing approaches for ETag functionality. - Suggests future improvements for ETag caching and metrics collection.

- Introduced a new test class `HandleEtagCacheInvalidationTest` to verify ETag behavior upon database updates. - Implemented tests to ensure ETag changes when the underlying data is modified and that the correct ETag is returned for subsequent requests. - Updated existing `HandleEtagTest` to include a test for returning 304 Not Modified when the ETag matches the client-provided value.

- Introduced logging for highly dynamic endpoints by tracking ETag history for each endpoint. - Implemented a mechanism to limit the number of tracked ETags and log when all tracked ETags are unique. - Updated documentation to reflect the new logging feature and its implications for performance optimization.

- Integrated `config/etag.php` for dynamic configuration of ETag functionality. - Added `enabled` and `log_dynamic_endpoints` flags to control feature behavior. This update improves flexibility and allows disabling ETag processing entirely when `enabled` is set to false.

- Added custom Trend metrics to measure and compare durations of 200 OK and 304 Not Modified responses - Validates that 304 responses are faster using If-None-Match header - Improved test clarity by focusing on ETag performance under load

- Implemented tracking of ETag values for specified endpoints - Added logic to identify dynamic endpoints when ETag history shows diff values

FOUR-20954: Monitor ETag Caching Performance

processmaker-sonarqube · 2025-01-23T16:16:36Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

eiresendez · 2025-01-13T17:51:34Z

Issue & Reproduction Steps

The server sends large amounts of data with each request, even if the data hasn’t changed. This causes unnecessary server load, increased response times and higher resource consumption, especially for repeated requests.

Solution

Added ETag caching headers to API endpoints.
The server will now respond with a 304 Not Modified when data remains unchanged, improving performance and reducing bandwidth usage.

How to Test

Verify that repeated requests for the same unchanged data return a 304 Not Modified status.
Check the docs/etag-caching.md for detailed steps and testing guidelines.

Related Tickets & Packages

FOUR-20929

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

- Introduced IfNoneMatch middleware to handle ETag-based caching: - Supports safe HTTP methods (GET, HEAD). - Returns "304 Not Modified" for matching ETags. - Handles multiple ETags and restores original request methods. - Added EtagManager for flexible ETag generation: - Default ETag generation uses MD5 hash. - Supports custom hash algorithms like SHA-256 via callbacks. - Included test cases: - Validate default ETag generation. - Test custom callback logic for ETag creation. This commit adds foundational support for ETag-based caching in the application.

…ware - Merged `etag.set` and `etag.if-none-match` middlewares into a single `etag.handle` middleware. - Simplified logic to reduce redundancy and improve maintainability. - Ensured ETag validation (`If-None-Match`) and generation are handled in the same flow. - Preserved compatibility with HEAD requests for consistency. This refactor improves clarity, reduces potential misconfigurations, and keeps the ETag logic centralized.

…d tests - Created tests for the new `HandleEtag` middleware: - Validates ETag generation and correctness. - Tests responses for both matching and non-matching `If-None-Match` headers. - Ensures proper handling of weak ETags (`W/`). - Removed old tests for `SetEtag` and `IfNoneMatch` middlewares as they are no longer needed. This commit improves test clarity and ensures the new ETag middleware behaves as expected.

- Added a test to validate ETag generation for user-specific routes using `etag:user`. - Simulates an authenticated user and verifies the ETag includes the user ID. These tests ensure the ETag middleware behaves correctly for both user-specific and common routes.

FOUR-20933: Implement ETag Caching for Screens Data

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

FOUR-20944: Implement ETag Caching for Task and Case Data

- Includes an overview of ETag functionality and benefits. - Provides implementation details for the HandleEtag middleware. - Covers ETag generation based on tables and response content. - Adds examples for conditional responses and route configuration. - Documents unit and manual testing approaches for ETag functionality. - Suggests future improvements for ETag caching and metrics collection.

- Introduced a new test class `HandleEtagCacheInvalidationTest` to verify ETag behavior upon database updates. - Implemented tests to ensure ETag changes when the underlying data is modified and that the correct ETag is returned for subsequent requests. - Updated existing `HandleEtagTest` to include a test for returning 304 Not Modified when the ETag matches the client-provided value.

- Introduced logging for highly dynamic endpoints by tracking ETag history for each endpoint. - Implemented a mechanism to limit the number of tracked ETags and log when all tracked ETags are unique. - Updated documentation to reflect the new logging feature and its implications for performance optimization.

- Integrated `config/etag.php` for dynamic configuration of ETag functionality. - Added `enabled` and `log_dynamic_endpoints` flags to control feature behavior. This update improves flexibility and allows disabling ETag processing entirely when `enabled` is set to false.

- Added custom Trend metrics to measure and compare durations of 200 OK and 304 Not Modified responses - Validates that 304 responses are faster using If-None-Match header - Improved test clarity by focusing on ETag performance under load

- Implemented tracking of ETag values for specified endpoints - Added logic to identify dynamic endpoints when ETag history shows diff values

FOUR-20954: Monitor ETag Caching Performance

processmaker-sonarqube · 2025-01-23T16:16:36Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

eiresendez · 2025-01-13T17:51:34Z

Issue & Reproduction Steps

The server sends large amounts of data with each request, even if the data hasn’t changed. This causes unnecessary server load, increased response times and higher resource consumption, especially for repeated requests.

Solution

Added ETag caching headers to API endpoints.
The server will now respond with a 304 Not Modified when data remains unchanged, improving performance and reducing bandwidth usage.

How to Test

Verify that repeated requests for the same unchanged data return a 304 Not Modified status.
Check the docs/etag-caching.md for detailed steps and testing guidelines.

Related Tickets & Packages

FOUR-20929

Code Review Checklist

I have pulled this code locally and tested it on my instance, along with any associated packages.
This code adheres to ProcessMaker Coding Guidelines.
This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
This solution fixes the bug reported in the original ticket.
This solution does not alter the expected output of a component in a way that would break existing Processes.
This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
This ticket conforms to the PRD associated with this part of ProcessMaker.

- Introduced IfNoneMatch middleware to handle ETag-based caching: - Supports safe HTTP methods (GET, HEAD). - Returns "304 Not Modified" for matching ETags. - Handles multiple ETags and restores original request methods. - Added EtagManager for flexible ETag generation: - Default ETag generation uses MD5 hash. - Supports custom hash algorithms like SHA-256 via callbacks. - Included test cases: - Validate default ETag generation. - Test custom callback logic for ETag creation. This commit adds foundational support for ETag-based caching in the application.

…ware - Merged `etag.set` and `etag.if-none-match` middlewares into a single `etag.handle` middleware. - Simplified logic to reduce redundancy and improve maintainability. - Ensured ETag validation (`If-None-Match`) and generation are handled in the same flow. - Preserved compatibility with HEAD requests for consistency. This refactor improves clarity, reduces potential misconfigurations, and keeps the ETag logic centralized.

…d tests - Created tests for the new `HandleEtag` middleware: - Validates ETag generation and correctness. - Tests responses for both matching and non-matching `If-None-Match` headers. - Ensures proper handling of weak ETags (`W/`). - Removed old tests for `SetEtag` and `IfNoneMatch` middlewares as they are no longer needed. This commit improves test clarity and ensures the new ETag middleware behaves as expected.

- Added a test to validate ETag generation for user-specific routes using `etag:user`. - Simulates an authenticated user and verifies the ETag includes the user ID. These tests ensure the ETag middleware behaves correctly for both user-specific and common routes.

FOUR-20933: Implement ETag Caching for Screens Data

…on to prevent ETag inconsistencies

- Process only GET and HEAD methods to ensure middleware relevance and avoid unnecessary processing for non-cacheable HTTP methods. - Add a check to determine if the response is cacheable, filtering out non-cacheable responses (e.g., those with 'no-store' directive or non-cacheable status codes). - Default ETag generation now includes user-specific data (auth()->id()) to enforce personalized caching by default. - Removed 'scope' and 'includeUser' logic for simplified and consistent caching behavior.

- Refactored `EtagManager` to support dynamic ETag generation based on configurable sources (`updated_at`). - Introduced `generateEtagFromTables` with a `source` parameter for flexibility in determining the source of truth. This update prepares the app for future scalability and allows switching between different ETag generation strategies.

- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

FOUR-20944: Implement ETag Caching for Task and Case Data

- Includes an overview of ETag functionality and benefits. - Provides implementation details for the HandleEtag middleware. - Covers ETag generation based on tables and response content. - Adds examples for conditional responses and route configuration. - Documents unit and manual testing approaches for ETag functionality. - Suggests future improvements for ETag caching and metrics collection.

- Introduced a new test class `HandleEtagCacheInvalidationTest` to verify ETag behavior upon database updates. - Implemented tests to ensure ETag changes when the underlying data is modified and that the correct ETag is returned for subsequent requests. - Updated existing `HandleEtagTest` to include a test for returning 304 Not Modified when the ETag matches the client-provided value.

- Introduced logging for highly dynamic endpoints by tracking ETag history for each endpoint. - Implemented a mechanism to limit the number of tracked ETags and log when all tracked ETags are unique. - Updated documentation to reflect the new logging feature and its implications for performance optimization.

- Integrated `config/etag.php` for dynamic configuration of ETag functionality. - Added `enabled` and `log_dynamic_endpoints` flags to control feature behavior. This update improves flexibility and allows disabling ETag processing entirely when `enabled` is set to false.

- Added custom Trend metrics to measure and compare durations of 200 OK and 304 Not Modified responses - Validates that 304 responses are faster using If-None-Match header - Improved test clarity by focusing on ETag performance under load

- Implemented tracking of ETag values for specified endpoints - Added logic to identify dynamic endpoints when ETag history shows diff values

FOUR-20954: Monitor ETag Caching Performance

processmaker-sonarqube · 2025-01-23T16:16:36Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

eiresendezand others added 25 commits November 26, 2024 14:09

Add ETag to Screen Responses
ea9795c

Merge branch 'epic/FOUR-20929' into story/FOUR-20933
924cd3c

Merge pull request#7771from ProcessMaker/story/FOUR-20933
c283de6
FOUR-20933: Implement ETag Caching for Screens Data

Merge branch 'release-2024-fall' into epic/FOUR-20929
7d33532

Exclude 'active_at' field from TaskResource and requestor serializati…
f058177
…on to prevent ETag inconsistencies

Implement ETag Caching for Task Responses
48cd7d6

Add ETag middleware to API group to support package routes
dae2468

Apply ETag middleware to v1.1 routes for consistent caching
8a94369

add ETag middleware to startProcesses route
cdaa80d
- Applied ETag middleware to the 'startProcesses' route for improved caching and reduced payload size. - Added default 'etag_tables' parameter set to 'processes' to optimize ETag generation for this route.

Merge pull request#7786from ProcessMaker/story/FOUR-20944
e64f7a7
FOUR-20944: Implement ETag Caching for Task and Case Data

Merge pull request#7817from ProcessMaker/story/FOUR-20950
24afdfc

Add ETag response time comparison for 200 vs 304
4e7afdc
- Added custom Trend metrics to measure and compare durations of 200 OK and 304 Not Modified responses - Validates that 304 responses are faster using If-None-Match header - Improved test clarity by focusing on ETag performance under load

Add ETag history tracking to detect dynamic endpoints
d590373
- Implemented tracking of ETag values for specified endpoints - Added logic to identify dynamic endpoints when ETag history shows diff values

Add long-duration test to validate ETag stability under sustained load
8086a78

Merge pull request#7834from ProcessMaker/story/FOUR-20954
c295567
FOUR-20954: Monitor ETag Caching Performance

eiresendez mentioned this pull request Jan 13, 2025
Implement eTag to the Modal for Cases #7790
Closed
9 tasks

eiresendez requested a review from estebangallego January 13, 2025 17:53

estebangallego approved these changes Jan 13, 2025
View reviewed changes

sanjacornelius added the 4.13.0 label Jan 13, 2025

ryancooley added the conflicts label Jan 23, 2025

Merge branch 'release-2025-winter' into epic/FOUR-20929
a6b7ee0

ryancooley merged commit 7ecbcd6 into release-2025-winterJan 23, 2025