Use HPS Loan IO for a second SDIO bus

My goal is to use an SDIO Host Controller IP to connect a wireless module to a Cyclone V SoC. I would like to use some HPS pins for this, so I have set them up as loan io.

I want to use HPS loan io pins for the following signals: SDIO CLK, SDIO IRQ, SDIO WP, SDIO CD. For the SDIO CMD and the SDIO DAT[3..0] signals, I have FPGA pins.

The problem is that I can't get the SDIO CLK to 25 MHz, all I'm getting is 300 kHz. Since the built-in SDIO bus is also using HPS pins, I assume it must be possible to reach a clock of 25 MHz on these pins. My question is: how?