Fix some load bugs #2384
Fix some load bugs #2384
Conversation
1. The concurrent running txn num of routine load job will be limited.
Limit by FE config 'max_running_txn_num_per_db'.
2. Create new routine load task when txn is VISIBLE, not after COMMITTED.
If publish version task has some error, there will be too many COMMITTED
txns in GlobalTransactionMgr.
| @@ -66,6 +67,10 @@ | |||
|
|
|||
| protected long timeoutMs = -1; | |||
|
|
|||
| // this status will be set when corresponding transaction's status is changed. | |||
| // so that user or other logic can know the status of the corresponding txn. | |||
| protected TransactionStatus txnStatus = TransactionStatus.UNKNOWN; | |||
There was a problem hiding this comment.
Yes, if it is set to COMMITTED, the timeout checker will not handle this task
There was a problem hiding this comment.
Yes, if it is set to COMMITTED, the timeout checker will not handle this task
| // 1. the number of running routine load tasks is limited by Config.max_routine_load_task_num_per_be | ||
| // 2. if we add routine load txn to runningTxnNums, runningTxnNums will always be occupied by routine load, | ||
| // and other txn may not be able to submitted. | ||
| if (runningRoutineLoadTxnNums.getOrDefault(dbId, 0) >= Config.max_running_txn_num_per_db) { |
There was a problem hiding this comment.
The task will not be renew before the txn is visible at above. So we do not limit the txn in here
There was a problem hiding this comment.
I prefer to add the double check here, in case we meet some other unexpected bugs.
There was a problem hiding this comment.
Maybe you are right, if I add this check here, may hurt some current routine load job user.
I will remove it.
|
For #2383, why there are always some committed but not visible txn before upgrade |
because publish task always meet error...which should be improved later.. |
For apache#2383 1. Limit the concurrent transactions of routine load job 2. Create new routine load task when txn is VISIBLE, not after COMMITTED. For apache#2267 1. All non-master daemon thread should also be started after catalog is ready. For apache#2354 1. `fixLoadJobMetaError()` should be called after all meta data is read, including image and edit logs. 2. Mini load job should set to CANCELLED when corresponding transaction is not found, instead of UNKNOWN.
For #2383
For #2267
For #2354
fixLoadJobMetaError()should be called after all meta data is read, including image and edit logs.of UNKNOWN.