Altera DDR3 traffic generator behavior question

My suggestion would be to "keep it simple" - if you find yourself in a position where you want to reset any of the components (controller or BIST, in this case) - I'd just reset everything. I can imagine asynchronously resetting just the BIST might cause issues for either the DDR3 controller or possibly parts of the generated Qsys interconnect in between the two components.

I have used the BIST by manually or automatically controlling it's reset signal to chain together tests, but I was always resetting it only when it's "done" status output was asserted (no active burst possible).

But back to your problem of how you could possibly be getting a timeout in the first place, I'm not sure why that could be. My inclination would be to use Signaltap and characterize the last transactions leading up to the timeout and first determine if there is a pattern to the failure or if they are random.