From Custom Instruction to Custom Peripheral

Some use multiple calls to the custom instruction to do what you are trying to do. Before going down that road will you be doing this same calculation over an entire frame of data? If so I recommend that you implement this as a hardware accelerator that can master the memory since it should be much more efficient (and the CPU can be doing something else in the meantime). You can also build your hardware to perform just the transform and use DMAs to shoved data in and out of your hardware.