Remove Queue.js by BobdenOs · Pull Request #233 · SAP/node-hdb

BobdenOs · 2024-04-05T17:39:57Z

We have a scenario in @cap-js/hana that a blob column is streamed out of the database and is being inserted into a different table. When doing this the Queue.js implementation creates a dead lock. As the lob read request cannot be send while actively processing the insert statement.

This PR solves this limitation by removing the Queue.js from the connection. Instead using the packetCount to identify what callback to use when a response is received from the the HANA system.

There is still a pseudo queue in the current implementation as simply sending the request to the system without waiting for an acknowledgement. Results in sporadic connection terminations by the HANA system. With the reason invalid packet length while sending the exact same packets.

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors Implementation Details To implement the "blocked" mode, the Queue is modified to a data structure which supports 3 operations 1. Push a task (preserving the order in which tasks were pushed) 2. Pop a task (remove the task in the order they were pushed) 3. Selective pop (remove a task in the order they were pushed given that the type matches a given variety) For optimization purposes, the tasks that can block are only READ_LOB requests. As such, we only require another readLobQueue which stores READ_LOB tasks to allow those to skip the queue. - An invariant of the Queue is that the queue must store the same READ_LOB tasks as the readLobQueue and possibly more READ_LOB tasks that are already run. If in the future, it is found that other tasks can block, it is possible to maintain a map<message type, list of queue tasks> and still have these 3 operations run in constant time, because the number of message types is finite (and small).

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors

BobdenOs · 2024-04-05T17:39:57Z

We have a scenario in @cap-js/hana that a blob column is streamed out of the database and is being inserted into a different table. When doing this the Queue.js implementation creates a dead lock. As the lob read request cannot be send while actively processing the insert statement.

This PR solves this limitation by removing the Queue.js from the connection. Instead using the packetCount to identify what callback to use when a response is received from the the HANA system.

There is still a pseudo queue in the current implementation as simply sending the request to the system without waiting for an acknowledgement. Results in sporadic connection terminations by the HANA system. With the reason invalid packet length while sending the exact same packets.

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors Implementation Details To implement the "blocked" mode, the Queue is modified to a data structure which supports 3 operations 1. Push a task (preserving the order in which tasks were pushed) 2. Pop a task (remove the task in the order they were pushed) 3. Selective pop (remove a task in the order they were pushed given that the type matches a given variety) For optimization purposes, the tasks that can block are only READ_LOB requests. As such, we only require another readLobQueue which stores READ_LOB tasks to allow those to skip the queue. - An invariant of the Queue is that the queue must store the same READ_LOB tasks as the readLobQueue and possibly more READ_LOB tasks that are already run. If in the future, it is found that other tasks can block, it is possible to maintain a map<message type, list of queue tasks> and still have these 3 operations run in constant time, because the number of message types is finite (and small).

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors

BobdenOs · 2024-04-05T17:39:57Z

We have a scenario in @cap-js/hana that a blob column is streamed out of the database and is being inserted into a different table. When doing this the Queue.js implementation creates a dead lock. As the lob read request cannot be send while actively processing the insert statement.

This PR solves this limitation by removing the Queue.js from the connection. Instead using the packetCount to identify what callback to use when a response is received from the the HANA system.

There is still a pseudo queue in the current implementation as simply sending the request to the system without waiting for an acknowledgement. Results in sporadic connection terminations by the HANA system. With the reason invalid packet length while sending the exact same packets.

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors Implementation Details To implement the "blocked" mode, the Queue is modified to a data structure which supports 3 operations 1. Push a task (preserving the order in which tasks were pushed) 2. Pop a task (remove the task in the order they were pushed) 3. Selective pop (remove a task in the order they were pushed given that the type matches a given variety) For optimization purposes, the tasks that can block are only READ_LOB requests. As such, we only require another readLobQueue which stores READ_LOB tasks to allow those to skip the queue. - An invariant of the Queue is that the queue must store the same READ_LOB tasks as the readLobQueue and possibly more READ_LOB tasks that are already run. If in the future, it is found that other tasks can block, it is possible to maintain a map<message type, list of queue tasks> and still have these 3 operations run in constant time, because the number of message types is finite (and small).

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors

BobdenOs · 2024-04-05T17:39:57Z

We have a scenario in @cap-js/hana that a blob column is streamed out of the database and is being inserted into a different table. When doing this the Queue.js implementation creates a dead lock. As the lob read request cannot be send while actively processing the insert statement.

This PR solves this limitation by removing the Queue.js from the connection. Instead using the packetCount to identify what callback to use when a response is received from the the HANA system.

There is still a pseudo queue in the current implementation as simply sending the request to the system without waiting for an acknowledgement. Results in sporadic connection terminations by the HANA system. With the reason invalid packet length while sending the exact same packets.

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors Implementation Details To implement the "blocked" mode, the Queue is modified to a data structure which supports 3 operations 1. Push a task (preserving the order in which tasks were pushed) 2. Pop a task (remove the task in the order they were pushed) 3. Selective pop (remove a task in the order they were pushed given that the type matches a given variety) For optimization purposes, the tasks that can block are only READ_LOB requests. As such, we only require another readLobQueue which stores READ_LOB tasks to allow those to skip the queue. - An invariant of the Queue is that the queue must store the same READ_LOB tasks as the readLobQueue and possibly more READ_LOB tasks that are already run. If in the future, it is found that other tasks can block, it is possible to maintain a map<message type, list of queue tasks> and still have these 3 operations run in constant time, because the number of message types is finite (and small).

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors

BobdenOs · 2024-04-05T17:39:57Z

We have a scenario in @cap-js/hana that a blob column is streamed out of the database and is being inserted into a different table. When doing this the Queue.js implementation creates a dead lock. As the lob read request cannot be send while actively processing the insert statement.

This PR solves this limitation by removing the Queue.js from the connection. Instead using the packetCount to identify what callback to use when a response is received from the the HANA system.

There is still a pseudo queue in the current implementation as simply sending the request to the system without waiting for an acknowledgement. Results in sporadic connection terminations by the HANA system. With the reason invalid packet length while sending the exact same packets.

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors Implementation Details To implement the "blocked" mode, the Queue is modified to a data structure which supports 3 operations 1. Push a task (preserving the order in which tasks were pushed) 2. Pop a task (remove the task in the order they were pushed) 3. Selective pop (remove a task in the order they were pushed given that the type matches a given variety) For optimization purposes, the tasks that can block are only READ_LOB requests. As such, we only require another readLobQueue which stores READ_LOB tasks to allow those to skip the queue. - An invariant of the Queue is that the queue must store the same READ_LOB tasks as the readLobQueue and possibly more READ_LOB tasks that are already run. If in the future, it is found that other tasks can block, it is possible to maintain a map<message type, list of queue tasks> and still have these 3 operations run in constant time, because the number of message types is finite (and small).

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors

BobdenOs · 2024-04-05T17:39:57Z

We have a scenario in @cap-js/hana that a blob column is streamed out of the database and is being inserted into a different table. When doing this the Queue.js implementation creates a dead lock. As the lob read request cannot be send while actively processing the insert statement.

This PR solves this limitation by removing the Queue.js from the connection. Instead using the packetCount to identify what callback to use when a response is received from the the HANA system.

There is still a pseudo queue in the current implementation as simply sending the request to the system without waiting for an acknowledgement. Results in sporadic connection terminations by the HANA system. With the reason invalid packet length while sending the exact same packets.

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors Implementation Details To implement the "blocked" mode, the Queue is modified to a data structure which supports 3 operations 1. Push a task (preserving the order in which tasks were pushed) 2. Pop a task (remove the task in the order they were pushed) 3. Selective pop (remove a task in the order they were pushed given that the type matches a given variety) For optimization purposes, the tasks that can block are only READ_LOB requests. As such, we only require another readLobQueue which stores READ_LOB tasks to allow those to skip the queue. - An invariant of the Queue is that the queue must store the same READ_LOB tasks as the readLobQueue and possibly more READ_LOB tasks that are already run. If in the future, it is found that other tasks can block, it is possible to maintain a map<message type, list of queue tasks> and still have these 3 operations run in constant time, because the number of message types is finite (and small).

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors

BobdenOs · 2024-04-05T17:39:57Z

We have a scenario in @cap-js/hana that a blob column is streamed out of the database and is being inserted into a different table. When doing this the Queue.js implementation creates a dead lock. As the lob read request cannot be send while actively processing the insert statement.

This PR solves this limitation by removing the Queue.js from the connection. Instead using the packetCount to identify what callback to use when a response is received from the the HANA system.

There is still a pseudo queue in the current implementation as simply sending the request to the system without waiting for an acknowledgement. Results in sporadic connection terminations by the HANA system. With the reason invalid packet length while sending the exact same packets.

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors Implementation Details To implement the "blocked" mode, the Queue is modified to a data structure which supports 3 operations 1. Push a task (preserving the order in which tasks were pushed) 2. Pop a task (remove the task in the order they were pushed) 3. Selective pop (remove a task in the order they were pushed given that the type matches a given variety) For optimization purposes, the tasks that can block are only READ_LOB requests. As such, we only require another readLobQueue which stores READ_LOB tasks to allow those to skip the queue. - An invariant of the Queue is that the queue must store the same READ_LOB tasks as the readLobQueue and possibly more READ_LOB tasks that are already run. If in the future, it is found that other tasks can block, it is possible to maintain a map<message type, list of queue tasks> and still have these 3 operations run in constant time, because the number of message types is finite (and small).

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors

BobdenOs · 2024-04-05T17:39:57Z

We have a scenario in @cap-js/hana that a blob column is streamed out of the database and is being inserted into a different table. When doing this the Queue.js implementation creates a dead lock. As the lob read request cannot be send while actively processing the insert statement.

This PR solves this limitation by removing the Queue.js from the connection. Instead using the packetCount to identify what callback to use when a response is received from the the HANA system.

There is still a pseudo queue in the current implementation as simply sending the request to the system without waiting for an acknowledgement. Results in sporadic connection terminations by the HANA system. With the reason invalid packet length while sending the exact same packets.

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors Implementation Details To implement the "blocked" mode, the Queue is modified to a data structure which supports 3 operations 1. Push a task (preserving the order in which tasks were pushed) 2. Pop a task (remove the task in the order they were pushed) 3. Selective pop (remove a task in the order they were pushed given that the type matches a given variety) For optimization purposes, the tasks that can block are only READ_LOB requests. As such, we only require another readLobQueue which stores READ_LOB tasks to allow those to skip the queue. - An invariant of the Queue is that the queue must store the same READ_LOB tasks as the readLobQueue and possibly more READ_LOB tasks that are already run. If in the future, it is found that other tasks can block, it is possible to maintain a map<message type, list of queue tasks> and still have these 3 operations run in constant time, because the number of message types is finite (and small).

Fixed the issue in SAP#233 where streaming blobs out of the database into another table can cause a deadlock - Added a "blocked" mode to the Queue which prevents tasks from running except for the blocking task and READ_LOB tasks - Modified ExecuteTask's run to free the queue while it waits for the Writer's getParameters - The callback of getParameters will enqueue the task again to send the packet - Before the freeing of the queue to the next task, ExecuteTask will block the queue to only allow itself and READ_LOB tasks to run - This prevents issues where exec's can run at the same time which will lead to HANA disconnecting and sending invalid LOB locator id errors

BobdenOs added 4 commits April 5, 2024 14:00