There are `python-matplotlib` and `python-numpy` packages in the Debian/Ubuntu repos.

However, if you want to run in a virtualenv (with no-site-packages), and pip install these packages from PyPI, you need some system dependencies installed first to build with:

$ sudo apt-get install build-essential python-dev libfreetype6-dev libpng-dev python-virtualenv

Then, you can create a virtualenv, and the installers for Numpy and Matplotlib will work: 

$ virtualenv env
$ cd env
$ source bin/activate
(env)$ pip install numpy matplotlib

...
...
Successfully installed numpy matplotlib
Cleaning up...