how I can create a custom instruction to manage operations on vectors

Easily :-)

Also (IIRC) qsys will directly expose the 2nd port of a memory block - so you can directly attach external VHDL to it.

Or you write your own Avalon slave interface for the memory block inside your VHDL.

In the latter case you may not be able to use the more efficient 'tightly coupled data' interface for the memory block (although that would also proclude DMA access to it).