You probably also want to arrange for the 'small data segment' to be in the on-chip memory and use section names ".sdata.*" so that the compiler will generate %gp relative addressing for the accesses.
Unfortunately the Altera builds of the compiler will not use %gp relative addressing for members of structures or arrays allocated to the 'small data' section. There are some gcc patches on the wiki that will make gcc use %gp relative addressing for structure members (constant offset from %gp), but not for arrays.
An alternative is to put all the global data inside a single structure (max 32kb) and use a global register variable to access it. gcc can generate better code for this than when using 'small data'. With care you can use %gp for the global register!