I bet you're not eating enough. I'm a fairly consistent eater (meaning in the work week I eat the same things a ton...chicken with some sort of herb rub or homemade sauce, along with tons of broccoli, for dinner most nights, etc) so one thing I did that helped me a ton was make a chart of everything I ate on a constant basis, both the good and the bad.
I included the item, a serving size, and then the calories, protein, fat and carbs for each item. It took me a couple hours, but it is an incredible finished product. I can now easily see how much food I really need to eat, and also what needs to belong in my weekly "cheat meal".
You can eat a ton of food and not gain (too much) fat as long as it is clean (protein, lean meats, healthy fats, complex carbs...limit fattier meats and simple carbs). Eat a lot of protein, especially. I've read a good goal is 1 gram of protein for every pound you weigh. Given my diet and that I usually have chicken for dinner as well as chicken deli meat in a sandwich (whole grain bread) at lunch, I can get to that goal fairly easily. I still like to supplement it with a protein shake immediately after I lift though so the protein can get in my bloodstream the quickest.
There are online calculators that give you an idea of what you need to eat, but instead of calorie counting every single thing I eat, I put together a couple meal plans using my nutrition chart. After seeing these meal plans I can easily tweak it around and know that I'm essentially more or less on track for the day.
If you are eating mostly clean, the amount of food is shocking and is actually (for me, at least) hard to eat in a day. So unless you are stuffing yourself full of clean foods to the point where you are eating even when you aren't hungry, you probably aren't getting enough haha!