To better understand the answer, let's first review some history.

Software development existed since the beginnings of the 1960s. The way software was created by that time was completely different and formal methodologies made sense at that time since they were created for that kind of development. A lot of developers were needed to write code and projects took a lot of time. It was expensive and only big companies could have the luxury to afford it. There were heavyweight methods.

But then, in the 1980s things changed. The PC was invented and new languages come to the playground. Now, you don’t need an army of software developers to create software nor big and expensive machines. The team now are a lot smaller since everything was simplified and soon everybody realizes that the old way to make software now is obsolete.

This was the realization by developers and during the 1990s, a number of lightweight software development methods evolved in reaction to the prevailing heavyweight methods like Rapid Application Development (RAD), dynamic systems development methods (DSDM), Scrum, Crystal, Extreme Programming (XP) and feature-driven development (FDD). Although these all originated before the publication of the Agile Manifesto, they are now collectively referred to as Agile software development methods.

In 2001, seventeen software developers met at a resort in Snowbird, Utah to discuss these lightweight development methods, including Kent Beck, Ward Cunningham, Dave Thomas, Jeff Sutherland, Ken Schwaber, Jim Highsmith, Alistair Cockburn, and Bob Martin. Together they published the Manifesto for Agile Software Development.

Currently the most common way to work is Scrum or Kanban and for the software development team they work great since it eliminates all the bureaucracy that previous formal methodologies had. For example, Rational Unified Process (RUP) from IBM, a current formal methodology still used, says that a lot of different roles and a lot of different processes are needed to develop software.

TSP/PSP are other formal methodologies created to be used together have fewer process and roles, and I actually love them, but they are heavily loaded with metrics.

On the other hand, Agile methodologies like Scrum eliminate all that and focus on software development by simplifying the software process, making software projects shorter with fewer people. This, for example, is the Scrum process.

That is the beauty of Agile methodologies: less time, less people and less difficulty. But it comes with a price.

Agile methodologies don’t have a lot of the safety features the formal methodologies have. Neglected and poor requirements/stories and developers with no experience make projects fail very easily.

Also since Agile methodologies don’t have a design phase, for long projects the lack of planning in this matter makes them, over time, more difficult to implement stories, since it needs a lot of code refactoring. The bigger the project gets, the more difficult it gets to change a lot of code to make a story fit.

A lot of people defend their right to not pay attention to this planning since Agile doesn’t encourage it explicitly, but they also don’t prohibited.

I have seen that borrowing good practices and models from the formal methodologies does wonders, but some of those who follow the Agile Orthodox see this as a sin. Once, I didn’t get a job because I said during the interview that I use a mixture of Agile and formal methodologies; the interviewer had the impression that I wasn’t qualified to lead a project.

So, I think Agile methodologies are great but you need to learn the principals that formal methodologies stand for, and improve the way you do your job even if you feel you are already good at it.

For big and complicated projects, I still recommend using formal methodologies since they lower a lot of the risk for a project to fail. A lot of people says they don’t work, but what they are really saying is that they aren't properly trained to work on those methodologies.