JMS : Java Message Service is an API that is part of Java EE for sending messages between two or more clients. There are many JMS providers such as OpenMQ (glassfish’s default), HornetQ(Jboss), and ActiveMQ.
RabbitMQ: is an open source message broker software which uses the AMQP standard and is written by Erlang.
JMS supports two models: one to one and publish/subscriber. RabbitMQ supports the AMQP model which has 4 models : direct, fanout, topic, headers.
JMS supports 5 different data types but RabbitMQ supports only the binary data type.
In AMQP, producers send to the exchange then the queue, but in JMS, producers send to the queue or topic directly.
JMS is specific for java users only, but RabbitMQ supports many technologies.
If you would like to know more about their performance, this benchmark is a good place to start, but look for others as well.