Slowdown when increasing the number of compute units

There is no general rule of thumb of global to local size.

My recommendation for you is to evaluate how to reduce your global memory bandwidth requirements as well as determine if the access pattern to global memory is efficient. For example if you load the same data from global memory multiple times attempt to store that data to local memory first so that you can re-use it onchip where there is more memory bandwidth. If your kernel doesn't access global memory sequentially that can also cause performance problems. For example this would be a good access pattern:

data = some_global_location [get_global_id(0)];

This would be a bad access pattern from a bandwidth perspective:

data = some_global_location [some_variable % 5];