It's psychological..haha..jk
It is difficult to dispute biomechanical evidence showing that there is forward rotation of the pelvis that leads to the flattening of the lumbar lordosis. As such, an increase in spinal compression occurs. High compression and shear forces for extended periods of time may result in pain and injury due to spinal creep and degeneration (sitting a few hours a day without back support, days on end). I'm not arguing that the stability ball isn't a good tool to improve core stability...but should only be used for that application, not as task seating. As a side note, any seating posture on any type of seating places greater stress to the spine than lying in the supine position or even standing up.
I recommend reviewing journal articles (ie. Dr. McGill from U of Waterloo, Chaffin from U of Michigan, etc) from reputable researchers in the field of spinal biomechanics before you come to your own conclusion. Those who have advocated stability balls as task seating either have a steel vertebrae or are trying to sell you a stability ball.
Sorry if I come across as 'stubborn', but I'm pretty adamant in improper usage of a stability ball.