As far as I see, the SignalTap specific Virtual JTAG protocol is undocumented. Scripting support allows an automation of the Quartus stp application, but low level access isn't discussed in the Quartus Software manual.
Also the DLL interface, that obviously exists in the Quartus software stack, is generally undocumented, except for a particular DLL dedicated to the software UART (jtag_atlantic.dll). It has been in discussed in the forum before.
I didn't yet feel a need to control SignalTap instances by external software, although I see a purpose. Accessing Source&Probe and In-System Memory Editor is more interesting as a general test tool, particularly for boards, that aren't equipped with a NIOS II or hardware processor. A documented interface would open this option to all Altera FPGA users. To my opinion, Altera should regard it as a cheap oportunity to improve the product versatility.