How do I know? Well, it seems people try agile, and it doesn't work for them. You can take a couple of examples from Ben Linders' post on InfoQ: "The Flexibility of Agile: Flaw or Strength?"
The first example explains that agile doesn't differentiate between large and small tasks, just by importance. This is completely wrong, of course. That’s not agile’s responsibility, it's the product owner's job to evaluate cost and value, and then decide.
Other misconceptions are that agile is an excuse to neglect proper design / planning / documentation / estimation / etc. This is, of course, false as well. Read the values of the Agile Manifesto, or go even deeper into XP and you'll see that's not true. We, “real” agile people, simply value other stuff more.
Agile is contextual, and matches an environment with many unknowns. If everything is known, a good planning ahead with a good change management process will probably work better. Of course, nothing is ever known completely, so traditional processes won't work.
This is my formal response, anyway.
But if agile is really contextual, how can I (or anyone else) say that it cannot be used as an excuse for bad design? In some situations, I actually saw that happening.
Everybody has an opinion
The main issue here is not the disillusionment with agile. People try it, in different flavors, and succeed or fail to a certain extent. That's not only acceptable, it's expected: I see agile as a process of learning, and on the way you'll have both.
The real failing of agile is what has failed countless methodologies before it. It grew, reshaped by interesting parties, and is now perceived as a dogmatic religion (actually a few of them). For a dogma, it's a binary choice: It's either the "It works" or "It fails" camp. (It wasn't always like that, by the way. Read the original manifesto and it’s language.)
Since a methodology can't work all the time, it can't be a silver bullet. Unfortunately, many people believe that's what agile is. And now, with their actual bad experience, they are taking shots at it.
And these failures are only half the story.
On the other side of the battlefield there people who know exactly what the failed practitioners have done wrong. "You misread the manifesto". "You should go back and understand what the Product Owner really does". "You misunderstood what estimation is".
And like any good religious war, everyone loses, including the original idea, that wasn't half bad.
The Good News
Not today. I warned about it a few years back. It’s a bad situation, and we’ve got ourselves to blame.
All I can ask of you is what I ask of myself: Keep experimenting until you find what works for you. Then, don't waste your time explaining "the right way" will work for everyone else.
Eliminate waste. That’s a good thing, right?