On March 11th, 2008, I gave e-conference presentation on dealing with large data in an Ajax application. The presentation can be seen by clicking here. It can be challenging to get your Ajax application to meet your data requirements.
The presentation walks through an explanation of which is faster to use XML vs. JSON. Included in the presentation was several sample applications that illustrate ways to page data using an Ajax application. Also available is a recording of the webinar with audio at Nexaweb.com.
XML vs. JSON
There was a lot of debate of which is better for data transfer. I ran a bunch of tests on data and it seems that for the most part JSON was faster than XML for data transfer. I have upgrade the performance test application to include tests for the XML and JSON parsing and Object Graph navigation. That being said, using data transfer objects and building the UI on the client isn't the fastest way. In the applications below, I create the UI on the server-side using PHP and then blend the HTML returned into the DOM based on location and blending parameter (append, replace, replace-children, insert-before, insert-after). This increased the amount of data the I could display.
Data Paging Techniques
Infinite Scrollbar (dzone.com)
dzone.com uses the infinite scrollbar pattern to fetch more articles when the user scrolls near the bottom of the visible area of the content window. This gives the impression of a continous stream of data. Because users don't need to consciously retrieve more data, people are likely to read more stories then they would if they had to click a button to get the next set of data. You will notice a lot of text to the left of the data window this is because getting the application to work correctly on all browsers was a "Bitch"; not the detection of the scroll position, but the scroll into view feature. Opera's calculation for where the element was was different from the other browsers and IE was left implemented because it would flicker up and down causing the browser to continously request more data.
Inifinite Scrollbar (Using the browser scrollbar)
The application uses the browsers outer scrollbar rather than one in a div. I think this use of the infinite scrollbar is more intuitive and easier to consumer the articles because they are not limited to the area of the div.
This application makes the user click a "More" button to fetch the next set of records. I actually liked this method for getting the data the best after implementing and using all the techniques, even though it required me to click the button. I could leave my mouse in the sample position and use the arrows to fetch the data, then when I got to the bottom, I just click the button.
I "leveraged" the paging bar that Google uses for the Analytics application. That paging bar makes the navigation of data that is properly ordered much easier. I can easily jump from page to page with a consist number of records in view. This technique is probably best for the fetching data that is used more for analytics rather than content reading.
The use of the "Infinite" scrollbar pattern is being used more and more in Ajax applications, but I am sure if this is the optimal way to retrieve data. There are a number of usability issues with this pattern when applied to applications that analyze data:
1.) Backwards and forward navigation of data is not implemented and would be weird if it was. What does a user do to get the last set of records they were viewing if they scrolled down. Yes I know scroll up, but how much the scroll bar become more sensitive the more data is fetched.
2.) My applications just fetched new data, but in applications that are displaying rows of data instead of content, telling the user where they are in the dataset increases the cost of the implementing the feature.
3.) Allowing for the users to be in the middle of the dataset without needing to fetch all the previous rows means increased implementation costs.
4.) In the simple implementation, navigation the data means you need to fetch all the data up to the point you want to see. This could be overcome but implementation costs goes up.
In order to get the application to be more seamless, I implemented each of them to automatically scroll to the beginning of the newly fetch dataset. This allowed the user to have continuity in the application. Without the auto-scrolling functionality, it was hard to figure out what items I had read and what items were new.
More User Interface work is needed in the area of data fetching and presentation, we have been doing it along time but there is so much that could be done. Talking with my co-worker Angel (not the vampire with the soul but the Ajax developer) he suggested using a software version of a scrollbar that instead of the browser scrollbar. You could use this to calculate acceleration and determine from that when to fetch data. You could also enhance the scrollbar to show what data has been collected already and where.
The dojotoolkit has a widget that uses a software scrollbar instead of the browser one. This could be enhanced as shown in the picture below. The green empty rectangles show you where you have data cache on the client-side and what record the scrollbar is over is denoted by the green transparent rectangles.