How does it work on multiple nodes?

[adampash]

Let's say I have a cluster of 5 different nodes, all running walex. What would happen in that scenario? Would they all trigger when an event was published? Do they round robin them? Something else more clever? Etc.?

Great project, appreciate it.

[cpursley]

Good question @adampash - let me do some testing and get back to you on this.

[cpursley]

It seems like no:

NOTICE 2024-01-12 12:13:54.034 [notice] Application services exited: Services.Application.start(:normal, []) returned an error: shutdown: failed to start child: WalEx.Supervisor
INFO    ** (EXIT) shutdown: failed to start child: WalEx.Replication.Supervisor
INFO        ** (EXIT) shutdown: failed to start child: WalEx.Replication.Server
INFO            ** (EXIT) :function_clause
INFOKernel pid terminated (application_controller) ("{application_start_failure,services,{{shutdown,{failed_to_start_child,'Elixir.WalEx.Supervisor',{shutdown,{failed_to_start_child,'Elixir.WalEx.Replication.Supervisor',{shutdown,{failed_to_start_child,'Elixir.WalEx.Replication.Server',function_clause}}}}}},{'Elixir.Services.Application',start,[normal,[]]}}}")
INFO
INFOCrash dump is being written to: erl_crash.dump...done

I'll start thinking about how to address this issue, thank you for bringing it to my attention.

[adampash]

Thanks @cpursley!

[krisleech]

When I start the app on multiple nodes I only get the events on one node, in my case this is okay, as I want the events in order and to only be processed once. On the other nodes I see:

[info] file=lib/walex/replication/server.ex  Replication slot changedatacapture_walex is active. Waiting for it to become inactive.

However when I terminate the node which has the replication slot another node attaches to the replication slot and receives all events, those already processed by the now terminated node.

This means I get the events more than once.

Also I assume the WAL is growing larger on database server.

I have durable_slots: true as I don't want to miss any events if the app goes down, or during deployment.

We are using lib_cluster, not that I think that makes any difference.

[cpursley]

Yeah, unfortunately I don't have a good solution yet for multiple nodes yet. Let me know what else you discover.