That's correct, for example if you have Nios II (2 masters), a DMA (2 masters), and some DDR SDRAM in your design. If you only want the DMA to be able to transfer data to/from DDR SDRAM then you don't connect the Nios II masters to that memory. The more masters you connect to Avalon slaves the larger the switch fabric becomes and the speed (fmax) may decrease as well. As a result this means that you should only connect masters to slaves where you really do need them.
BTW if you want some additional help, the Nios forum is full of SOPC Builder users.