It could be a good idea to split all your data in chars before filling the buffer. The code will be a bit slower, but at least you won't depend on big/little endian problems and other subtleties.
AFAIK the behaviour of C code writing ints in a char buffer isn't defined, and can change from one platform to another, or even from a compiler to another (and, with a little help from Murphy's law, maybe even from a version of a compiler to another).
I'm not even sure that defining your packet data as a struct will give predictable results, but I may be wrong on this one.