I have solved this problem. I built a PCIe endpoint with Avalon memory-mapped interface as .ip, rather than as a .qsys. Quartus allows this (and it seems to be fine for cores with the Avalon streaming interface), but it does not provide a way to program the BARs, i.e. to set the size of each addressable memory region. As a result, the BAR2_SIZE_MASK parameter is set to 0, and the upper index of rxm_address_reg[BAR2_SIZE_MASK -1:0] is a negative number (- 1). This causes the compile error.
The solution is to build a .qsys macro which contains just the PCIe core, and export the Avalon-MM master interfaces through bridges.