.qsys files (and directories containing .qsys files) can definitely be added to the .ipx file and be picked up by Qsys.
The .ipx can be used a couple different ways, but for you I think the easiest thing would be to simply use it to specify the search path. Assuming that "Top_Level" is where your project is located and where Qsys will run from, try creating a user_components.ipx file that looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<library>
<path path="Subsystem_X/*" />
<path path="Subsystem_X/IP/**/*" />
</library>
What that says is to first look in Subsystem_X (to pickup the .qsys subsystem there), and then the second line says to search all the subdirectories of "IP" (to pickup your components).
You can actually avoid creating the .ipx file in your toplevel directory if you place your "Subsystem_X" directory inside an 'ip' directory, since it is in the default search path, but you would need to create a .ipx in the subdirectory since the search will stop recursing when it finds your .qsys subsystem. For example,
/Top_Level/ip/Subsystem_X/subsystem_X.qsys
/Top_Level/ip/Subsystem_X/subsystem_X_components.ipx
/Top_Level/ip/Subsystem_X/IP/Component_A/Component_A_hw.tcl
/Top_Level/ip/Subsystem_X/IP/Component_B/Component_B_hw.tcl
I believe with that structure, the "subsystem_X_components.ipx" you could create with ip-make-ipx, telling it to index your Subsystem_X/IP directory.
My suggestion would be to start with the simplest .ipx above to get oriented, and then get more complicated (more modular) from there.
As far as debugging, you can leave Qsys running and use File->Refresh system to cause it to re-search. In the dialog box that pops up, you can scroll up and see where it has attempted to process the directories in your .ipx file and it will report counts of components, so you can have some hint that it is or is not finding your files.