Autoscroll stops when list item is removed from the DOM

[gorhawk]

When I click the middle button on the mouse and move the indicator (i think that is called autoscrolling), it works fine until the list item I clicked on gets scrolled out of the 'buffer' and is removed from the DOM. Then the scroll just stops.
However, it works fine if I middle click on the scrollbar.
I found it really weird trying to produce the minimal example that could reproduce the issue because I have this problem everywhere. I'm using webpack and babel to transpile.

<Infinite
    containerHeight={this.state.scrollContainerHeight}
    elementHeight={this.elementHeight}
>
  {listItems}
</Infinite>

List items are like:
<div draggable="true" tabIndex="0">some text in anchor tag...</div>

Any tips on debugging this?