Repeated CL kernel invocation hangs after some successful runs during clFinish

It's me again and I'm facing a new issue on my way trying to use OpenCL for realtime DSP on an FPGA ;)

A quick description of my setup:

I've set up a custom BSP for my Terasic DE10 Standard. I've added a Terasic THDB_ADA AD/DA converter board to my FPGA board and created a custom Qsys component to use it. The signal path looks like this

ADA Card --> Custom Qsys Block --> Avalon Dual-Clock FIFO IP Core --> Output of FIFO exposed as IO channel to CL Kernel through BoardSpecs.xml modification --> CL kernel that gets invoked over and over and simply copies blocks of samples from channel to global memory.

My host code (using the cl C++ bindings) runs this loop on a high priority thread

while (true)
        {
            rxTxKernel.setArg (0, rxSamplesB.getCLBuffer());
 
            if (threadShouldExit())
                break;
 
            // let the kernel work on buffer B while the host processes buffer A
            queue.enqueueTask (rxTxKernel);
            currentCallback->processRFSampleBlock (rxSamplesA, txDummyBuf);
            queue.finish();
            
            // switch the buffers
            rxTxKernel.setArg (0, rxSamplesA.getCLBuffer());
 
            if (threadShouldExit())
                break;
 
            // let the kernel work on buffer A while the host processes buffer B
            queue.enqueueTask (rxTxKernel);
            currentCallback->processRFSampleBlock (rxSamplesB, txDummyBuf);
            queue.finish();
}

The kernel currently simply copies 65536 samples from the channel to global memory.

This runs for approx 100 times successfully and then suddenly stops forever at queue.finish() (e.g. clFinish under the hood). I can then see the FIFO feeding the channel running full, as I'm displaying this via an LED on the board, so this can not be the kernel waiting because of a channel stall.

Any idea what this could be?