I sorta went through the OP's scenario a few times with exes, that introduces an additional angle, besides the lingering emotional aspect that others here have touched upon. In my case they were the ones who wanted me to be the FWB after we had drifted apart as BFs. In fact, at one point I was in bed with 2 of them together, a 3-way one of them arranged.
I'd agree to sex with them during dry spells, nothing else going on, and I guess the same was true for them. I don't have much will power when a guy's inviting me into bed, especially someone who isn't a complete stranger.
But as soon as I had a new BF the FWB arrangement would cease. Believe it or not, despite being a happily slutty single, when I'm in a serious relationship I become 100% monogamous & faithful, no cheating ever.
And so these exes would get upset when the FWB deal would abruptly stop. I'd still be comfortable with them as friends (though both my partners have NOT liked that), but the sex was out. One ex simply wouldn't take no for an answer, I think he deliberately wanted to cause a break-up with my partner. He even went so far as to try to seduce me while my partner was sleeping in our bedroom not 15 feet away (the ex was in the guest room following his midnight plane arrival, staying in town until we could drive him home next morning).
Therefore I see potential problems down the road when you start a FWB relationship with an ex, should he not want to leave the scene gracefully when a new BF comes along. Of course that can be true of any FWB, but I think the ex-BF presents the greater risk of that situation developing. If none of you are going to be monogamous anyway then I suppose it's not a problem, and then you're just left with the question of whether you can break emotionally from a guy with whom you're having sex.