MMC failing on linux-altera on CycloneV/Atlas-SoC, while ltsi kernel is not

I'm setting up a Linux system on a CycloneV on Atlas-SoC, and I'm using Yocto Pyro to build. I'm using meta-altera from https://github.com/kraj/meta-altera to build the kernel. The preloader and u-boot is built using Quartus 17.1 and EDS. I'm using meta-linaro and use Linaro GCC 6.3.

If I select KERNEL_PROVIDER="linux-altera-ltsi" which is now at 4.1.33, everything boots fine. However if I select KERNEL_PROVIDER="linux-altera" and version 4.12 or 4.11, I get failures in MMC, which ultimately fails the boot.

Any ideas to why?

     mmc_host mmc0: Bus speed (slot 0) = 200000000Hz (slot req 25000000Hz, actual 25000000HZ div = 4)
     mmc0: new SD card at address 9a6c
     mmcblk0: mmc0:9a6c SU02G 1.84 GiB 
      mmcblk0: p1 p2 p3
     mmcblk0: error -110 transferring data, sector 136328, nr 264, cmd response 0x900, card status 0x0
     mmc_host mmc0: Bus speed (slot 0) = 200000000Hz (slot req 400000Hz, actual 400000HZ div = 250)
     mmc_host mmc0: Bus speed (slot 0) = 200000000Hz (slot req 25000000Hz, actual 25000000HZ div = 4)
     mmcblk0: error -110 transferring data, sector 136328, nr 264, cmd response 0x900, card status 0x0
     mmcblk0: retrying using single block read
     mmcblk0: error -110 transferring data, sector 136337, nr 255, cmd response 0x900, card status 0x0