Projection fault behavior changes after projection is reset

[antonmos]

Describe the bug

1. publish an event of type `test' to any stream
2. publish an event to a stream $et-test managed by the $by_event_type projection2.
   • I am aware that we SHOULD NOT write events to streams managed by projections.
3. publish another event of type test to any stream
4. Observe that this causes the $by_event_type projection to fault
5. Invoke reset command on $by_event_type projection
6. This rebuilds all of the $et-* streams.
7. repeate step 1. The event correctly appears in the $et-test
8. repeat step 2
9. observe that the projection does not fault
10. observe that event from step 7 disapeared from $et-test

See gist for code that demonstrates this

Expected behavior
The behavior before and after the reset should be the same.
That said, the behavior after the reset actually seems beneficial - i.e. events erroneously written to projection-managed streams are discarded and projections does not fault.

Actual behavior
The behavior changed after the reset

Config/Logs/Screenshots
If applicable, please attach your node configuration, logs or any screenshots.

EventStore details

• EventStore server version: 26.10.0

• Operating system: Linux

• EventStore client library and version (if applicable):

Additional context
Add any other context about the problem here.

[linear]

DB-794 Projection fault behavior changes after projection is reset

[ylorph]

Reproduced and got the follwing logs.
It seems the engine is recovering by itself the second time and event is added directly to $et-* and afterwards an event is added that should update the previous one

eventstoredb  |    at EventStore.Projections.Core.Services.Processing.Emitting.EmittedStream.<>c__DisplayClass53_0.<SubmitListEvents>b__0(ReadStreamEventsBackwardCompleted completed) in /home/runner/work/TrainStation/TrainStation/EventStore/src/EventStore.Projections.Core/Services/Processing/Emitting/EmittedStream.cs:line 341
eventstoredb  |    at EventStore.Core.Helpers.ReadStreamEventsBackwardHandlers.AdHoc.Handle(ReadStreamEventsBackwardCompleted message) in /home/runner/work/TrainStation/TrainStation/EventStore/src/EventStore.Core/Helpers/ReadStreamEventsBackwardHandlers.cs:line 87
eventstoredb  |    at EventStore.Core.Helpers.ReadStreamEventsBackwardHandlers.Tracking.Handle(ReadStreamEventsBackwardCompleted message) in /home/runner/work/TrainStation/TrainStation/EventStore/src/EventStore.Core/Helpers/ReadStreamEventsBackwardHandlers.cs:line 45
eventstoredb  |    at EventStore.Core.Messaging.RequestResponseDispatcher`3.Handle(TResponse message) in /home/runner/work/TrainStation/TrainStation/EventStore/src/EventStore.Core/Messaging/RequestResponseDispatcher.cs:line 92
eventstoredb  |    at EventStore.Core.Messaging.RequestResponseDispatcher`3.EventStore.Core.Bus.IHandle<TResponse>.Handle(TResponse message) in /home/runner/work/TrainStation/TrainStation/EventStore/src/EventStore.Core/Messaging/RequestResponseDispatcher.cs:line 82
eventstoredb  |    at EventStore.Core.Bus.IHandle`1.EventStore.Core.Bus.IAsyncHandle<T>.HandleAsync(T message, CancellationToken token) in /home/runner/work/TrainStation/TrainStation/EventStore/src/EventStore.Core/Bus/IHandle.cs:line 19
eventstoredb  | --- End of stack trace from previous location ---
eventstoredb  |    at EventStore.Core.Bus.InMemoryBus.MessageTypeHandler`1.InvokeAsync(Message message, CancellationToken token) in /home/runner/work/TrainStation/TrainStation/EventStore/src/EventStore.Core/Bus/InMemoryBus.MessageHierarchy.cs:line 130
eventstoredb  |    at EventStore.Core.Bus.InMemoryBus.DispatchAndWatchSlowMsg(MessageTypeHandler handlers, Message message, CancellationToken token) in /home/runner/work/TrainStation/TrainStation/EventStore/src/EventStore.Core/Bus/InMemoryBus.cs:line 81
eventstoredb  |    at EventStore.Core.Bus.QueuedHandlerThreadPool.System.Threading.IThreadPoolWorkItem.Execute() in /home/runner/work/TrainStation/TrainStation/EventStore/src/EventStore.Core/Bus/QueuedHandlerThreadPool.cs:line 143
eventstoredb  | [    1,15,13:44:58.672,ERR] ProjectionManager              The '$by_event_type' projection faulted due to 'The given checkpoint is invalid. Possible causes might include having written an event to the projection's managed stream. The bad checkpoint: Phase: 0'
eventstoredb  | [    1,16,13:45:25.830,INF] ProjectionManager              Resetting '$by_event_type' projection
eventstoredb  | [    1,31,13:45:25.850,INF] ProjectionManager              '$by_event_type' projection source has been written
eventstoredb  | [    1,26,13:47:22.188,INF] CoreProjectionCheckpointManager Failed to write events to stream $et-test. Error: WrongExpectedVersion
eventstoredb  | [    1,26,13:47:22.189,INF] ProcessingStrategySelector     Projection '$by_event_type'(787f3901-471e-406a-84d8-c6afb40eb588) restart has been requested due to: 'The '$et-test' stream has been written to from the outside. Expected Version: 4, Current Version: 5. Checkpoint: C:0/P:-1.'

[ylorph]

WE need to check if it resets from start or last successful checkpoint