verilog-module instantiation

Everytime I've done that, it's because I forgot to drive something into the module, it became unknown and propogated around. You said you're driving everything, but maybe worth a double-check.

ya, confirmed that i drive all the inputs into the module.

Is the output dependent on something that might not be initialized like a flip flop or RAM in module B?

module A;
  reg  s, sq, sqe;
  reg  q;
  reg  s_length, q_length, sqe_length;
  reg subset_start;
  wire subset_out,subset_done;
  
  
  B mod_B(q,q_length,sqe,sqe_length,subset_start,subset_out,subset_done);
always @*
  begin
    subset_start=0;                             
    cS=3'd 1;
    
    s=s_in;
    s_length=5'd 1;
    
    q=s_in;
    q_length=5'd 1;
    sq={s,q};
    sqe=sq>>1;
    sqe_length=s_length+q_length-1;
    subset_start=1;
(some codes here)
case (subset_out)
      1: begin
.
.
.
(some codes here)
end
     0:begin
.
.
.
(some codes here)
end
end
endmodule
module B(q,q_length,sqe,sqe_length,start,out,done);
  
  input start;
  input  q_length;
  input  q;
  input  sqe;
  input  sqe_length;
  output reg out,done;
always @ (posedge start)
  begin
.
.
.
(codes)
end
endmodule

I meant initialize registers/RAMs in the submodule. I see in module B you have some sort of net with sensitivity to 'start' but without seeing that net I have no clue what you have implemented. If your intent was to create a flipflop you should be driving it with a clock and reset as well and using an enable bit if you don't want it to capture every clock cycle. If you don't drive a reset into a register at the beginning of the simulation you'll get unknown data out of it (Modelsim can't assume high or low outputs from unitialized regs/RAMs). There are templates in Quartus II under the edit menu you can take a look at for typical hardware blocks such as registers, memory, etc...

As shown above, module A has no in- or ouputs. But there are undefined signals like s_in. I think. it's pointless to discuss the problem details at this level of information.

actually module A do have inputs and outputs. I just didn't include them here because my main concern is on the module instantiation. I just want to show that the input to module B are defined. So just ignore the inputs and output of module A.

You mean that module B is pure combinational. That's possible of course.

Inputs need to be driven in the simulation by respective source variables. Verilog variables (registers) have an intial state of 'x'.