In a marriage (or any partnership) when one member embarks on major life and lifestyle changes, it can be stressful if the other retains the old way of life. That goes for more than fitness (which is synonymous with looks and desirability to some, but not all) - if one goes from a homebody to a more active social life, for instance, while the other wants to stay home like they both did before ... well, this can be a test. More than that, it can be a sign that you're actually growing apart.
Growth is good. And no relationship exists in a frozen state - not a healthy one. Change is part of the equation, even if it's not visible.
One thing to remember is that the person "getting fit" also is experiencing higher energy - he may wish you'd join him in that new level of "power" and energy, and he may find it hard to sit still so much (truly!). Also, part of their regime almost always means a change of diet - are you two parting ways there? Sometimes the guy on the regime feels his partner is sabotaging his efforts by keeping the same old foods around ...
So there are lots of issues. And attractiveness or sexiness is often the least of it - it's the other aspects that can cause more trouble.
I'm just getting back into the habit of regular exercise myself, after a few years of slipping, gradually. And at my age it is hard to stay motivated - when you're a kid you can undo 6 months of damage in a few weeks. In your 50s you can undo 6 months of effort in the same couple of weeks!
I suggest meeting him half way - you may not have the same unthusiasm for "working out", but a little bit won't kill you - it might keep you alive!
(I note a misspelling above, but I'm keeping it. It's my new word: unthusiasm - what you muster up for something you really don't want to do, but want to be able to know you've done.)