-
-
Notifications
You must be signed in to change notification settings - Fork 62
check duplicate message before enqueue #260
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Hello @sonus21, |
rqueue-core/src/main/java/com/github/sonus21/rqueue/core/impl/RqueueMessageEnqueuerImpl.java
Outdated
Show resolved
Hide resolved
if(isMessageAlreadyExist(queueName, messageId)){ | ||
log.warn("trying to enqueue duplicate message {}", messageId); | ||
return false; | ||
} | ||
return enqueue(queueName, messageId, message); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But this is not really atomic: if 2 same messageId
are enqueued at the same time using this method, then both may pass the check and both will be enqueued.
I have checked how uniqueness is implemented in db-scheduler.
Basically, the logic looks like this:
try {
if (already queued)
return false
insert to queue // (!!!) id is unique in DB schema, so the query will fail if it tries to insert duplicate
} catch (DB exception) {
if (already queued) // second check of "already queued"
return false
else
throw exception
}
Can we do something like this here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's not required here, because redis is a single threaded unlike database.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, but this method does at least 2 separate Redis operations.
If 2 messages are scheduled at the same time, you can't guarantee it will be "isMessageAlreadyExist -> enqueue -> isMessageAlreadyExist -> enqueue", but not "isMessageAlreadyExist -> isMessageAlreadyExist -> enqueue -> enqueue".
Or can you?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
potentially, we can.. i've updated the pr to use setIfAbsent redis command
Description
Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes: #259
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Test Configuration:
Checklist: