Hand tracking

I am trying to detect hand in video. I have implemented background subtraction but i have problem in tracking the hand by bounding box.When i try to make a box around hand a small noise appear on frame and make my bounding box get wrong dimension. For getting a box dimension i search for white Pixel in each frame then save the max and min X and Y dimension then draw the bounding box.

Please help me how to ignore these kind of island noise.