Description
PublishInSequence
- Support an optional parameter to make the specific
Channel-Topic
completely blocking. It means if you send msg1 to a specificChannel-Topic
then another client sends msg2 to the sameChannel-Topic
before msg1 finish, will receive an error likeChannel-Topic
is busy.
like:
PublishInSequence(msg *Message, orderKey string, option.IsChanelTopicBlocking) error
- Waiting
ACK
signal before the message that in consumer side is done.
We probably need to implement acknowledgment waiting methods for both of these approaches.
[Example 1]
Send msg1 and receive it's result
Send msg2 and receive it's result
[Example 2]
Send msg1 and msg2 together
Receive result of msg1 first until then msg2 result wait
for blocking feature in publish side.
Use polling to obtain the status until the consumer consumes successfully or fails to jump out of the poll.
we need an another function to check the current status of job.
- After the service crashes suddenly and restarts, the publishing side can still wait for the "ACK" signal.
the example scenario:
Manju is using beanq and call sequential publish function to execute some DB update with a pid supplied by cart and wait for the ACK. Then suddenly that Manju pod crashed. Then latter, how cart confirm the status of that pid? because the pid generated by cart and Manju crash so cart client will timeout as well. Therefore, cart will retry using same pid to update db again but task probably done by the beanq and the ack still inside beanq memory and no one read that ack yet.
- How to split the
channel-topic
into smaller granules because the scenario of one execution after another in a sequential queue limits the number of messages in thechannel-topic
from being too many, otherwise it will cause message backlog.
solution1:
Thinking about the smallest granules or atom.
How to notify which
channel-topic
should be subscribed in all scenarios?
radix tree?
notify the consumer through another queue with only the topic name?
OR
solution2:
1. Either publisher blocks the publish by checking the `channel-topic` status
2. Consumer workflow will do that by getting the message from `channel-topic` based on the status
OR
Messages are put into the queue according to groups. Each group will complete a complete business process. The execution between groups is asynchronous, but the messages within the group must be executed synchronously and sequentially. If the consumer gets the group messages, it will use the workflow method to execute or rollback. The key to simplifying things is the concept of grouping. This will prevent the queue from being blocked, but it can also persist the message to the queue.
example:
publish: send -> msg group{1: add, 2: save, 3:confirm } -> queue
consumer: subscribe -> msg group -> do workflow 1: add, 2: save 3: confirm -> done
workflow
rollback in the workflow processing
https://github.com/dtm-labs/dtm
https://github.com/harshadmanglani/polaris
Implementing workflow mechanism on the consumer side.