Custom Loop and Infinite Scroll (by Jetpack)

Hi Community,

I’m looking for a workaround on using infinite scroll from jetpack with a custom loop. I’ve set a query to only show posts of categories A at the index page.
If the infinite scroll is triggered, sometimes it happens, that posts are duplicated. I’ve already find out, what the problem is, but have no idea how to resolve it. Maybe some one of you has a hint for me.

The issue: It seems that the infinite scroll loop has a fixed starting position (paged 2) but sets paged variable differently than the index loop.

Posts in database:

Max number to show on front page: 3
Custom Loop for front page only showing category A.
Result: A1, A4, A5 -> everything is correct

On scroll, infinite loop is triggered.
Infinite Loop starts with paged 2 (as max number on front page is set on 3, the loop also counts B2, B3 and sets A4 as the first entry on the new page:
Result A4 (duplicate), A5 (duplicate) , A6 -> duplicate post

The attached image illustrates the issue.

Do you have an idea how to solve this?