I think one has to consider exactly what is meant by gifted. Is the primary criterion that they are doing fine, but could do harder work and aren't being challenged by the normal curriculum? Or are we going more for a classification type of definition - they are children who will actually fare poorly in school because they have different learning styles and something innate to their giftedness causes the standard curriculum to not "work" for them? In the former case, I think the solution is to go back to tracked classes. The modern concept of "differentiated learning" is really just a code word for conducting multiple tracks within the classroom which is a ridiculous expectation to put on a teacher with a class of more than several children. To me, this is not a money issue, a "neglecting the gifted issue" - it's political correctness and the self esteem movement once again running amok.
The second case is more intriguing to me. If gifted children are, among other things, characterized by an inability to flourish within the standard curriculum, then I think we have to ask ourselves some questions before we can determine whether we are neglecting them: Do they benefit in the long run from a specialized program? That is, does being in such a program have benefits that extend into adulthood, leading to intellectual development and successes that would not have occurred without the program? Or, once they are confronted with a more traditional setting (work, college, etc.), do these individuals simply flounder again?
Ideally, all children should be challenged to the maximum of their abilities. I say ideally, because I think we've taken on a financially crippling burden of expectation in some cases - I know there are people who would disagree vociferously, but I don't see the point of spending tens of thousands of dollars yearly on out of district placement for an individual who has a profound level of MR, for example, because at the end of the day, what education has been accomplished? I'd apply the same criterion to providing gifted education - it can't just feel good now and keep the kids out of the teacher's hair. There has to be some long term benefit.