Auction and Bidding Applications with Document-Based NoSQL

before we can do that, we need to figure out how we are going to use them. as i mentioned, i am going to use ebay as the source for “application mockups”.  so i went to ebay and took a couple of screen shots.

here is the actual auction page:

and here is the actual bids page.

this tells us several things:

• bids aren’t really accessed for the main page.
• there is a strong likelihood that the number of bids is going to be small for most items (less than a thousand).
• even for items with a lot of bids, we only care about the most recent ones for the most part.

this is the auction document as we have last seen it:

{
   "quantity":15,
   "product":{
      "name":"flying monkey doll",
      "colors":[
         "blue & green"
      ],
      "price":29,
      "weight":0.23
   },
   "startsat":"2011-09-01",
   "endsat":"2011-09-15"
}

the question is where are we putting the bids? one easy option would be to put all the bids inside the auction document, like so:

{
   "quantity":15,
   "product":{
      "name":"flying monkey doll",
      "colors":[
         "blue & green"
      ],
      "price":29,
      "weight":0.23
   },
   "startsat":"2011-09-01",
   "endsat":"2011-09-15",
   "bids": [
     {"bidder": "bidders/123", "amount": 0.1, "at": "2011-09-08t12:20" }
   ]
}

the problem with such an approach is that we are now forced to load the bids whenever we want to load the auction, but the main scenario is that we just need the auction details, not all of the bids details. in fact, we only need the count of bids and the winning bid, it will also fail to handle properly the scenario of high interest auction, one that has a lot of bids.

that leave us with few options. one of those indicate that we don’t really care about bids and auction as a time sensitive matter. as long as we are accepting bids, we don’t really need to give you immediate feedback. indeed, this is how most auction sites work. they give you a cached view of the data, refreshing it every 30 seconds or so. the idea is to reduce the cost of actually accepting a new bids to the minimum necessary. once the auction is closed, we can figure out who actually won and notify them.

a good design for this scenario would be a separate bid document for each bid, and a map/reduce index to get the winning bid amount and big count. something like this

     {"bidder": "bidders/123", "amount": 0.1, "at": "2011-09-08t12:20", "auction": "auctions/1234"}
     {"bidder": "bidders/234", "amount": 0.15, "at": "2011-09-08t12:21", "auction": "auctions/1234" }
     {"bidder": "bidders/123", "amount": 0.2, "at": "2011-09-08t12:22", "auction": "auctions/1234" }

and the index:

from bids in docs.bids
select new { count = 1, bid.amount, big.auction }

select result from results
group result by result.auction into g
select new 
{
   count = g.sum(x=>x.count),
   amount = g.max(x=>x.amount),
   auction = g.key
}

as you can imagine, due to the nature of ravendb’s indexes, we can cheaply insert new bids, without having to wait for the indexing to work. and we can always display the last calculated value of the auction, including what time it is stable for.

that is one model for an auction site, but another one would be a much stringer scenario, where you can’t just accept any bid. it might be a system where you are charged per bid, so accepting a known invalid bid is not allowed (if you were outbid in the meantime). how would we build such a system? we can still use the previous design, and just defer the actual billing for a later stage, but let us assume that this is a strong constraint on the system.

in this case, we can’t rely on the indexes, because we need immediately consistent information, and we need it to be cheap. with ravendb, we have the document store, which is acidly consistent. so we can do the following, store all of the bids for an auction in a single document:

{
   "auction": "auctions/1234",
   "bids": [
     {"bidder": "bidders/123", "amount": 0.1, "at": "2011-09-08t12:20", "auction": "auctions/1234"}
     {"bidder": "bidders/234", "amount": 0.15, "at": "2011-09-08t12:21", "auction": "auctions/1234" }
     {"bidder": "bidders/123", "amount": 0.2, "at": "2011-09-08t12:22", "auction": "auctions/1234" }
    ]
}

and we modify the auction document to be:

{
   "quantity":15,
   "product":{
      "name":"flying monkey doll",
      "colors":[
         "blue & green"
      ],
      "price":29,
      "weight":0.23
   },
   "startsat":"2011-09-01",
   "endsat":"2011-09-15",
   "winningbidamount": 0.2,
   "bidscount" 3
}

adding the bidscount and winningbidamount to the auction means that we can very cheaply show them to the users. because ravendb is transactional, we can actually do it like this:

using(var session = store.opensession())
{
  session.advanced.optimisticconcurrency = true;
  
  var auction = session.load<auction>("auctions/1234")
  var bids = session.load<bids>("auctions/1234/bids");
  
  bids.addnewbid(bidder, amount);
  
  auction.updatestatsfrom(bids);
  
  session.savechanges();
}

we are now guaranteed that this will either succeed completely (and we have a new winning bid), or it will fail utterly, leaving no trace. note that addnewbid will reject a bid that isn’t the higher (throw an exception), and if we have two concurrent modifications, ravendb will throw on that. both the auction and its bids are treated as a single transactional unit, just the way it should.

the final question is how to handle high interest auction, one that gather a lot of bids. we didn’t worry about it in the previous model, because that was left for ravendb to handle. in this case, since we are using a single document for the bids, we need to take care of that ourselves. there are a few things that we need to consider here:

• bids that lost are usually of little interest.
• we probably need to keep them around, just in case, nevertheless.

therefor, we will implement splitting for the bids document. what does this means?

whenever the number of bids in the bids document reaches 500 bids, we split the document. we take the oldest 250 bids and move them to historical bids document, and then we save.

that way, we have a set of historical documents with 250 bids each that no one is ever likely to read, but we need to keep, and we have the main bids document, which contains the most recent (and relevant bids. a high interest auction might end up looking like:

• auctions/1234 <- auction document
• auctions/1234/bids <- bids document
• auctions/1234/bids/1 <- historical bids #1
• auctions/1234/bids/2 <- historical bids #2
  

and that is enough for now i think, this post went on a little longer than i intended, but hopefully i was able to explain to you both the final design decisions and the process used to reach them.

thoughts?