How do I tell whether I should implement the machine I want to build as a field programmable gate array?

1) First thing to do is define your machine in detail and write the VHDL for it. Then run it in a simulation environment to see if it really does what you intend, functionally and performance-wise. You can do this more or less independent of the implementation technology, FPGA or ASIC.

2) Then once it is working in simulation, you can size/find an FPGA to act as a first trial implementation. If you just want a lab validation vehicle FPGA is the way to go. Even for low to medium production volumes FPGA is still your best bet.

3) Then if it is wildly successful and you can convince venture capitalists to front you several million dollars you can decide to go for an ASIC suitable for high volume production (like more than a million units a year). Until you do 1 and 2 forget about 3.