If you declare it at a file scope then the contents will basically be stored in a ROM. File scope __constant arrays need to be preinitialized so that the contents can be baked into the programming file.
If you declare it as a kernel argument then a read-only cache will be implemented so that contents from SDRAM when read get cached by the __constant cache. If you are familar with instruction caches of a CPU it's a lot like that. This __constant cache can be warmed up because the contents will only become evicted back out to SDRAM if the host overwrites that values between kernel invocations. Here is an example of how this can be handy.
Let's say you have a FIR filter and you will keep the coeffients constant but you don't want to hardcode them at the file scope. If you call the kernel multiple times across different NDRanges you can move the coefficients down to the FPGA before invoking the kernel the first time, then avoid copying the coefficients for subsequent calls of the same kernel. Between these kernel launches the cache should remain warmed up and the coeffiecients ready to go for each kernel call after the first one. Of course this assumes the __constant cache is large enough to hold the data you want to have cached, by default it's 16kB but you can override this using a flag to pass in a new size (remember the value is in bytes, and I recommend powers of 2 for the cache size).