Unit Testing Named Queues: Spring 3+maven2+Google App Engine

Problem, you have a task that you know can take more than 30 seconds to complete, what do you do?  What if this task needs to be triggered every day at a specific time?  Google provides several mechanisms to to solve just this problem, queues and scheduled task, respectively.

Queues
First, let's explore the "queue" system implemented in GAE.  Note, at this time, the "queue" is still experimental which means that the API can change around some, so make sure you have strong unit testing, but don't worry, we'll cover that a bit further down the page.  To implement queues, you need to let the web app know that your expecting queues to be used, and to do this, you need to create a file named queue.xml in your WEB-INF folder (example: /src/main/webapp/WEB-INF/queue.xml).  The queue.xml file has a specific layout and that will not be covered in depth in this post.  Please read up on the queue.xml structure on Google's documentation.  We will expand on the knowledge since the documentation is not complete.

Scheduled Task
Scheduled task are task that will trigger by GAE at repeatable times, emulating linux cron, or Quartz application.  To implement a cron task, simply create a file called cron.xml and place it in your WEB-INF folder (example:/src/main/webapp/WEB-INF/cron.xml).  This post will not go into detail on the implementation of scheduled task so please check Google's documentation to get a firm understanding before reading on. In order to not pack too much into one post, Scheduled Task will be covered in the next post.

Our Problem
The scenario we're going to explain is: You have an birthday tracking software and you want a user's friends to be notified, via email, that their friend is having a birthday.  Assume that the code for the services named are tested and fully working.

To solve this we will use a queue and the scheduled task system provided by Google App Engine.

First, let's tackle the queue portion, then we will create the scheduled task.  Google's documentation on unit testing queues is very limited and only shows how to retrieve the "default" queue.  Since we could have multiple task in our system that need to be run at different rates, we will want to create a new Queue.  Let's open (or create) the queue.xml file and insert the following:

<?xml version="1.0" encoding="UTF-8"?>
<queue-entries>
    <queue>
        <name>birthdayemail</name>
        <rate>10/s</rate>
        <bucket-size>10</bucket-size>
    </queue>
</queue-entries>

Next, let's create a simple service (Note: assume that the Autowired resources are working and imports are included at the top of the file, this has been reduced to emphasize the solution)

@Service("birthdayService")
public class BirthdayServiceImpl implements BirthdayService {

    // this is the same name as in your queue.xml file
    private static final String EVENT_REMINDER_TASK_NAME = "birthdayemail";
    
    @Autowired
    private BirthdayDAO birthdayDAO;

    public List handleGuestEmailsForBirthday(Date date) {
        if (date == null) {
            throw new IllegalArgumentException("Date can not be null");
        }
        List<Guest> guestBirthdays = birthdayDAO.getByDate(date);
        for (Guest guest : guestBirthdays) {
            String TASK_URL = "/queue/birthday-emails/" + guest.getId()
            final TaskOptions taskOptions = TaskOptions.Builder.url(TASK_URL);
            // create a unique task name, note, must conform to [a-z] regex
            taskOptions.taskName(guest.getName() + "Task");
            taskOptions.method(TaskOptions.Method.POST);

            Queue queue = QueueFactory.getQueue(EVENT_REMINDER_TASK_NAME);
            // commented out to show the default queue shown in the docs
            //Queue queue = QueueFactory.getDefaultQueue();
            queue.add(taskOptions);
        }
        return events;
    }
}

Let's look at what's here:
First, we have to create a URL that will be run when the task is implemented. This URL will handle the unit of work specified, and should run in under 30 seconds per GAE's restrictions. We will not cover what this servlet does, just assume it uses the ID given in the URL path, pulls up the Guest, then uses that to get the Guest's friends' emails; then uses that to construct an email and pass to an EmailService. (HINT: the EmailService could also implement a queue so each email is separated out to run individually)

Next, notice that we are implementing a "Task Name". The this is optional, but does help when debugging to figure out what task is failing or what is running.

Now we're on to the TaskOptions. TaskOptions are a helper function used to combine URL and model data to pass on to the servlet handling the queue. Just a suggestion, but to follow the REST ideals, setting the RequestMethod to POST or PUT is advisable, depending on what you are trying to do. All Task should be idempotent, meaning that if the task fails or is interrupted, the task can be run again and not harm the data.

Lastly, we have the QueueFactory retrieving the Named Queue (the same name in the queue.xml file). Simply add your taskOptions object to the queue.add and move on.

Unit Testing Named Queues
Next, let's setup a unit test to test our named queue.  The Google documentation does not explicitly explain how to properly setup a test case for one, datastore aware context and named queries.  Below is an example of a test case written to test our Named Queries.  (Note, as before, imports have been removed and the services used have been used before.  This is not intended to be the end-all-be-all for unit testing GAE, please refer to previous post for more help, and yes, you might be able to use JMock or Mockito instead of persisting data, but I have not explored the option).

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"/testApplicationContext.xml"})
public class QueueBirthdayServiceTest
        extends AbstractJUnit4SpringContextTests {

    @Autowired
    private BirthdayService birthdayService;
    private final LocalServiceTestConfig[] configs = {
        new LocalDatastoreServiceTestConfig(),
    /* NOTE: THE QUEUE XML PATH RELATIVE TO WEB APP ROOT, More info below */
        new LocalTaskQueueTestConfig()
                .setQueueXmlPath("src/test/resources/test-queue.xml")};
    private final LocalServiceTestHelper helper
            = new LocalServiceTestHelper(configs);

    @Before
    public void setUp() {
        helper.setUp();
    }

    @After
    public void tearDown() {
        helper.tearDown();
    }

    @Test
    public void testHandleBirthdayQueueEmails() throws InterruptedException {
        // setup data
        Date birtdayDate = DateUtils.parse("10/16/1900");
        // assume this builds the data for guest/guest's friends
        int guestWithBD = 5; // # of birthdays on date
        int gPerBirthday = 10; // # of emails being sent per birthday
        setupDataForBirthdays(birthdayDate, guestWithBD, gPerBirthday);

        List<Guest> birthdayOnDay = birthdayService.
                handleGuestEmailsForBirthday(birthdayDate);
        assertEquals(guestWithBD, birthdayOnDay.size());
        // pause for a moment to allow queue to fill from previous statment
        Thread.sleep(1000);
        // verify # of birthdays with that day's expire date
        LocalTaskQueue ltq = LocalTaskQueueTestConfig.getLocalTaskQueue();
        final Queue queue = QueueFactory
                .getQueue(BirthdayService.EVENT_REMINDER_TASK_NAME);
        //final Queue queue = QueueFactory.getDefaultQueue();
        QueueStateInfo qsi = ltq.getQueueStateInfo()
                .get(queue.getQueueName());
        assertNotNull(qsi);
        int expectedTaskCount = guestWithBD*gPerBirthday;
        assertEquals(expectedTaskCount, qsi.getTaskInfo().size());
        assertEquals(birthdayOnDay.get(0).getID() + "Task",
                qsi.getTaskInfo().get(0).getTaskName());

    }
}

 NOTE: If the syntax or unit testing is very foreign to you, please visit my previous post on Unit Testing on Google App Engine.

Let's explore more
The first half is not too exiting, just setting up data and calling the service that creates the queue.  We are creating 5 guest with birthdays on the given date, and each of those 5 guest have 10 friends who we intend to email.  This service is already "written" out (above)

What you need to pay attention to is at the top of the file, during the setup of the "helper", we have constructed it with a LocalDatastoreServiceTestConfig and LocalTaskQueueTestConfig objects. The second, the "LocalTaskQueueTestConfig" is the important one to add when using a queue. If you are not using the DefaultQueue, you will need to explicitly state where the queue.xml file is. I suggest that you create a test-queue.xml file and place it in your /src/test/resources folder, so as to not mix production data and testing. NOTE: This file is loaded based relative to the ROOT application folder. The rest of the test should be pretty self explanatory.

In the next post, we will uncover how to wire the service up to a Scheduled Task so you can automate the emails being sent. Stay tuned!

From http://www.ensor.cc/2010/11/unit-testing-named-queues-spring.html