I wouldn't necessarily eliminate the dairy products. A 2004 study in Obesity Research showed showed that for a group of people consuming the same total calories and fat, a high dairy diet was actually linked to greater weight loss than a low dairy diet, with a calcium supplemented diet falling partway between the two. The sample size was admittedly small (initial article n=32 individuals, with several follow ups showing the same results), and the subject may not have been representative of society as a whole (all began the study obese and had previously been rather low in calcium), but the high dairy diet group showed statistically significant increases in both total weight loss, and total fat loss as determined by dual x ray measurements. Further, a significantly greater percentage of the high dairy group's fat loss was trunk fat, and calcium supplementation alone didn't fully transform an otherwise low-dairy diet into the same results.
Overall, a little yogurt, low fat milk, or hard cheese in your diet might actually increase the amount of fat you can lose from your trunk.