Routine Load may takes too many FE memory

[morningman]

Describe the bug
When there are many routine load jobs and the config max_running_txn_per_db is low,
some of routine load tasks may be accumulated.

I am not sure if this is the reason of high memory consumption. I will trace it later.

[morningman]

The reason is that beIdToBatchTask in RoutineLoadTaskScheduler is not cleaned. And more and more TRoutineLoadTask objects are added to this map.

It is not cleaned because the submitBatchTasksIfNotEmpty() is not called because the following conditions are not met:

if (needScheduleTasksQueue.size() == 0 || idleSlotNum == 0) {
 submitBatchTasksIfNotEmpty();
}

This can happen when we set FE config max_running_txn_num_per_db smaller than
(num of BE * max_concurrent_task_num_per_be).
Routine load tasks always failed to begin transaction due to exceed max_running_txn_num_per_db limit and be pushed back to the needScheduleTasksQueue.

So needScheduleTasksQueue is always non-empty, but there are idle slots available.

[morningman]

Mainly fix the following issues:

1. A null pointer exception is raised when a database or table is dropped. The expected behavior is that the routine load job is stopped.
2. Memory leaks. Batch routine load task submissions are no longer performed, and modifications are submitted separately for each task.
3. Unreasonable task timeout.

Routine load tasks should not be queued in the BE thread pool for execution. The task sent to the BE should be executed immediately, otherwise the task in the FE will be timeout first. Eventually leads to constant timeout for all subsequent tasks.

4. All routine load job should be scheduled once it being submitted. Not waiting the available BE slot. Otherwise, all later submitted jobs may not be scheduled forever.