--- Quote Start ---
How is it possible to create a custom instruction interface (?) in an IP core? I think in your design you plan to have such an interface (if my understanding is correct)
I have read the Avalon interface manual (for sopc) and I do not see any interface of that sort.
--- Quote End ---
Yes I see the details are "light" to "non-existent" on how to do it via TCL.
Probably the easiest way is to use Qsys (File->New Component) and then from the component editor use the Templates menu to add the type of instruction you would like to add.
For what it's worth, I personally like editing the TCL manually, so I would just get the tool to generate the first round of port definitions and then copy&paste that into my own .tcl file.
(I have had trouble with the Component Editor in the past, and I do not like it).