Skip to content

[FEAT] Recording for features #39

Open
@zhongdalu-trechina

Description

@zhongdalu-trechina

PublishInSequence

  1. Support an optional parameter to make the specific Channel-Topic completely blocking. It means if you send msg1 to a specific Channel-Topic then another client sends msg2 to the same Channel-Topic before msg1 finish, will receive an error like Channel-Topic is busy.
    like:
    PublishInSequence(msg *Message, orderKey string, option.IsChanelTopicBlocking) error
  2. Waiting ACK signal before the message that in consumer side is done.
    We probably need to implement acknowledgment waiting methods for both of these approaches.
[Example 1]
Send msg1 and receive it's result
Send msg2 and receive it's result

[Example 2]
Send msg1 and msg2 together
Receive result of msg1 first until then msg2 result wait

for blocking feature in publish side.
Use polling to obtain the status until the consumer consumes successfully or fails to jump out of the poll.

we need an another function to check the current status of job.

  1. After the service crashes suddenly and restarts, the publishing side can still wait for the "ACK" signal.
    the example scenario:

Manju is using beanq and call sequential publish function to execute some DB update with a pid supplied by cart and wait for the ACK. Then suddenly that Manju pod crashed. Then latter, how cart confirm the status of that pid? because the pid generated by cart and Manju crash so cart client will timeout as well. Therefore, cart will retry using same pid to update db again but task probably done by the beanq and the ack still inside beanq memory and no one read that ack yet.

  1. How to split the channel-topic into smaller granules because the scenario of one execution after another in a sequential queue limits the number of messages in the channel-topic from being too many, otherwise it will cause message backlog.

solution1:
Thinking about the smallest granules or atom.

How to notify which channel-topic should be subscribed in all scenarios?

radix tree?
notify the consumer through another queue with only the topic name?

OR

solution2:

1. Either publisher blocks the publish by checking the `channel-topic` status
2. Consumer workflow will do that by getting the message from `channel-topic` based on the status

OR

solution3:
image

Messages are put into the queue according to groups. Each group will complete a complete business process. The execution between groups is asynchronous, but the messages within the group must be executed synchronously and sequentially. If the consumer gets the group messages, it will use the workflow method to execute or rollback. The key to simplifying things is the concept of grouping. This will prevent the queue from being blocked, but it can also persist the message to the queue.

example:

publish: send -> msg group{1: add, 2: save, 3:confirm } -> queue
consumer: subscribe -> msg group -> do workflow 1: add, 2: save 3: confirm -> done 

workflow

rollback in the workflow processing
https://github.com/dtm-labs/dtm
https://github.com/harshadmanglani/polaris

Implementing workflow mechanism on the consumer side.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions