That constraint will be in the shell, it might be an argument count limit, or a byte length limit. Try searching the source tree for the error message!
The kernel will have its own limit - but that need not apply to shell builtins.
The 'Too many filename matches' error might even be coming from the part of the shell that expands the wildcard - rather than part of the attempt to run a program. If the error is returned by "set -- *.dat" then you probably wont get the shell to expand the list anywhere.
That means you'll need to use something else to generate the list of names, 'find' or 'ls .' will work - with the latter you'll need to filter the unwanted names yourself.
You also won't want to run 'rm' for every file - unless you add it as a shell builtin - because of the fork+exec costs.
You might actually be able to use: find . -name '*.dat' -delete