When you die, you go to sleep and never wake up. When you are born you wake up but never remember going to sleep.
So then, if you die you go to sleep, and you have dreams about your life and other things, and since your brain forgets a lot of things so it needs to be refreshed. When you die, you dream of your life, about heaven or something like that, and people dream of other people they never met. And dreaming takes time so what if you remember all your past lives dreams.
You die, dream of your life and past lives, keep getting refreshed on these dreams but not knowledge, you forget everything but your dreams, and after some time you are reincarnated, not into the first birth, but a random one, you could be reincarnated a million years after death or a few seconds after death.