DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

The Latest Data Engineering Topics

article thumbnail
The “4+1” View Model of Software Architecture
In November 1995, while working as Lead software architect at Hughes Aircraft Of Canada Philippe Kruchten published a paper entitled: "Architectural Blueprints—The “4+1” View Model of Software Architecture". The intent was to come up with a mechanism to separate the different aspects of a software system into different views of the system. Why? Because different stakeholders always have different interest in a software system. Some aspects of a system are relevant to the Developers; others are relevant to System administrators. The Developers want to know about things like classes; System administrators want to know about deployment, hardware and network configurations and don't care about classes. Similar points can be made for Testers, Project Managers and Customers. Kruchten thought it made sense to decompose architecture into distinct views so stakeholders could get what they wanted. In total there were 5 views in his approach but he decided to call it 4 + 1. We'll discuss why it's called 4 + 1 later! But first, let's have a look at each of the different views. The logical view This contains information about the various parts of the system. In UML the logical view is modelled using Class, Object, State machine and Interaction diagrams (e.g Sequence diagrams). It's relevance is really to developers. The process view This describes the concurrent processes within the system. It encompasses some non-functional requirements such as performance and availability. In UML, Activity diagrams - which can be used to model concurrent behaviour - are used to model the process view. The development view The development view focusses on software modules and subsystems. In UML, Package and Component diagrams are used to model the development view. The physical view The physical view describes the physical deployment of the system. For example, how many nodes are used and what is deployed on what node. Thus, the physical view concerns some non-functional requirements such as scalability and availability. In UML, Deployment diagrams are used to model the physical view. The use case view This view describes the functionality of the system from the perspective from outside world. It contains diagrams describing what the system is supposed to do from a black box perspective. This view typically contains Use Case diagrams. All other views use this view to guide them. Why is it called the 4 + 1 instead of just 5? Well this is because of the special significance the use case view has. When all other views are finished, it's effectively redundant. However, all other views would not be possible without it. It details the high levels requirements of the system. The other views detail how those requirements are realised. 4 + 1 came before UML It's important to remember the 4 + 1 approach was put forward two years before the first the introduction of UML which did not manifest in its first guise until 1997. UML is how most enterprise architectures are modelled and the 4 + 1 approach still plays a relevance to UML today. UML 2.0 has 13 different types of diagrams - each diagram type can be categorised into one of the 4 + 1 views. UML is 4 + 1 friendly! So is it important? The 4 + 1 approach isn't just about satisfying different stakeholders. It makes modelling easier to do because it makes it easier to organise. A typical project will contain numerous diagrams of the various types. For example, a project may contain a few hundred sequence diagrams and several class diagrams. Grouping diagrams of similar types and purpose means there is an emphasis in separating concerns. Sure isn't it just the same with Java? Grouping Java classes of similar purpose and related responsibilities into packages means organisation is better. Similarly, grouping different components into different jar files means organisation is better. Modelling tools will usually support the 4 + 1 approach and this means projects will have templates for how to split the various types of diagrams. In a company when projects follow industry standard templates again it means things are better organised. The 4 + 1 approach also provides a way for architects to be able to prioritise modelling concerns. It is rare that a project will have enough time to model every single diagram possible for an architecture. Architects can prioritise different views. For example, for a business domain intensive project it would make sense to prioritise the logical view. In a project with high concurrency and complex timing it would make sense to ensure the process view gets ample time. Similarly, the 4 + 1 approach makes it possible for stakeholders to get the parts of the model that are relevant to them. References: Architectural Blueprints—The “4+1” View Model of Software Architecture Paper http://www.cs.ubc.ca/~gregor/teaching/papers/4+1view-architecture.pdf Learning UML 2.0 by Russ Miles & Kim Hamilton. O'Reilly From http://dublintech.blogspot.com/2011/05/41-view-model-of-software-architecture.html
December 28, 2011
by Alex Staveley
· 53,932 Views
article thumbnail
How to deploy a neo4j instance in Amazon EC2 in 10 minutes
Neo4j is a high-performance, NOSQL graph database with all the features of a mature and robust database. In this post I will explain how to deploy a neo4j instance in Amazon EC2 web service. For this tutorial to take you no more than 10 minutes you should be able to execute properly some bash commands like mv, tar, ssh and scp (secure copy). I also assume that you have an account in Amazon Web Services and you are familiar to the process of launching instances. If not, I strongly recommend you to follow this starting guide and complete it till you manage to connect to your instance with ssh. Start downloading the latest stable version of neo4j. Which you can find here. The “Community Edition” fits well for development purposes. Do not forget to select the Unix version of the server. This will download a tar.gz file which you will copy to your EC2 instance later. While you download the neo4j server open the AWS Management Console and launch a Basic 32-bit Amazon Linux AMI. If you want to launch an Ubuntu AMI please notice that it doesn’t ship with Java, which is required for running neo4j. If you are not familiar with key pairs, pem files or security groups I insist you to follow the EC2 starting guide I mentioned above. You can either create a new security group or use the default, but you will need to configure a new security rule for the neo4j server port. After launching the instance, create a TCP rule on port 7474 with source 0.0.0.0/0. Here you are opening port 7474 for anyone. If you are planning to use the neo4j REST API and remotely call it from another server, for example a Rails application hosted in Heroku, for security reasons, you may want to change the source field to the address of your Heroku server. Do not forget to open port 22 (SSH), this is typically the first rule normal people create after launching an instance. You are almost done! You should now install neo4j in your instance. Open a terminal in your localhost and navigate to the path where you downloaded neo4j. Copy the file to your Amazon instance by using the scp command: scp -i your_pem_file.pem neo4j-community-1.6.M01-unix.tar.gz ec2-user@YOUR_PUBLIC_INSTANCE_DNS:/home/ec2-user Please notice that you will need to change the path to your pem file, typically placed in ~/.ssh, the filename of the neo4j server you just downloaded and the plublic DNS of your instance. Now connect to your instance with SSH: ssh -i your_pem_file.pem ec2-user@YOUR_PUBLIC_INSTANCE_DNS Untar the neo4j server: tar xvfz neo4j-community-1.6.M01-unix.tar.gz.tar.gz Move it to /usr/local and rename the folder to neo4j: sudo mv neo4j-community-1.6.M01 /usr/local/neo4j Almost done!!! You should now open neo4j-server.properties under the conf directory and add the following line: org.neo4j.server.webserver.address=0.0.0.0 This lines allows anyone to connect remotely to your neo4j database server. Now run the start script. From the neo4j server folder. sudo ./bin/neo4j start Finally, open a browser and access the webadmin interface of your neo4j database by typing http://YOUR_PUBLIC_INSTANCE_DNS:7474. You should see the Neo4j Monitoring and Management Tool, pretty cool! If not, ask me You can now try using the REST API and the curl bash command to insert nodes and relationships. I hope this post helped you, good luck! Follow me on Twitter @negarnil Source: http://www.cloudtmp.com/java/how-to-deploy-a-neo4j-instance-in-amazon-ec2-in-10-minutes/
December 27, 2011
by Nicolas Garnil
· 27,532 Views · 1 Like
article thumbnail
How to order by multiple columns using Lambas and LINQ in C#
Today, I was required to order a list of records based on a Name and then ID. A simple one, but I did spend some time on how to do it with Lambda Expression in C#. C# provides the OrderBy, OrderByDescending, ThenBy, ThenByDescending. You can use them in your lambda expression to order the records as per your requirement. Assuming your list is “Phones” and contains the following data public class Phone { public int ID { get; set; } public string Name { get; set; } } public class Phones : List { public Phones() { Add(new Phone { ID = 1, Name = "Windows Phone 7" }); Add(new Phone { ID = 5, Name = "iPhone" }); Add(new Phone { ID = 2, Name = "Windows Phone 7" }); Add(new Phone { ID = 3, Name = "Windows Mobile 6.1" }); Add(new Phone { ID = 6, Name = "Android" }); Add(new Phone { ID = 10, Name = "BlackBerry" }); } } If you were to use LINQ Query , the query will look like the one below dataGridView1.DataSource = (from m in new Phones() orderby m.Name, m.ID select m).ToList(); Simple isn’t it ? It very simple using Lamba expression too. Your Lambda’s expression for the above LINQ query will look like the one below dataGridView1.DataSource = new Phones().OrderByDescending(a => a.Name).ThenByDescending(a => a.ID).ToList();
December 24, 2011
by Senthil Kumar
· 137,089 Views
article thumbnail
Enabling JMX in Hibernate, Ehcache, Quartz, DBPC and Spring
A collection of short how-to's for enabling JMX in several popular Java technologies. Continuing our journey with JMX (see: ...JMX for human beings) we will learn how to enable JMX support (typically statistics and monitoring capabilities) in some popular frameworks. Most of this information can be found on project's home pages, but I decided to collect it with few the addition of some useful tips. Hibernate (with Spring support) Exposing Hibernate statistics with JMX is pretty simple, however some nasty workarounds are requires when JPA API is used to obtain underlying SessionFactory class JmxLocalContainerEntityManagerFactoryBean() extends LocalContainerEntityManagerFactoryBean { override def createNativeEntityManagerFactory() = { val managerFactory = super.createNativeEntityManagerFactory() registerStatisticsMBean(managerFactory) managerFactory } def registerStatisticsMBean(managerFactory: EntityManagerFactory) { managerFactory match { case impl: EntityManagerFactoryImpl => val mBean = new StatisticsService(); mBean.setStatisticsEnabled(true) mBean.setSessionFactory(impl.getSessionFactory); val name = new ObjectName("org.hibernate:type=Statistics,application=spring-pitfalls") ManagementFactory.getPlatformMBeanServer.registerMBean(mBean, name); case _ => } } } Note that I have created a subclass of Springs built-in LocalContainerEntityManagerFactoryBean. By overriding createNativeEntityManagerFactory() method I can access EntityManagerFactory and by trying to downcast it to org.hibernate.ejb.EntityManagerFactoryImpl we were able to register Hibernate Mbean. One more thing has left. Obviously we have to use our custom subclass instead of org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean. Also, in order to collect the actual statistics instead of just seeing zeroes all the way down we must set the hibernate.generate_statistics flag. @Bean def entityManagerFactoryBean() = { val entityManagerFactoryBean = new JmxLocalContainerEntityManagerFactoryBean() entityManagerFactoryBean.setDataSource(dataSource()) entityManagerFactoryBean.setJpaVendorAdapter(jpaVendorAdapter()) entityManagerFactoryBean.setPackagesToScan("com.blogspot.nurkiewicz") entityManagerFactoryBean.setJpaPropertyMap( Map( "hibernate.hbm2ddl.auto" -> "create", "hibernate.format_sql" -> "true", "hibernate.ejb.naming_strategy" -> classOf[ImprovedNamingStrategy].getName, "hibernate.generate_statistics" -> true.toString ).asJava ) entityManagerFactoryBean } Here is a sample of what can we expect to see in JvisualVM (don't forget to install all plugins!): In addition we get a nice Hibernate logging: HQL: select generatedAlias0 from Book as generatedAlias0, time: 10ms, rows: 20 EhCache Monitoring caches is very important, especially in application where you expect values to generally be present there. I tend to query the database as often as needed to avoid unnecessary method arguments or local caching. Everything to make code as simple as possible. However this approach only works when caching on the database layer works correctly. Similar to Hibernate, enabling JMX monitoring in EhCache is a two-step process. First you need to expose provided MBean in MBeanServer: @Bean(initMethod = "init", destroyMethod = "dispose") def managementService = new ManagementService(ehCacheManager(), platformMBeanServer(), true, true, true, true, true) @Bean def platformMBeanServer() = ManagementFactory.getPlatformMBeanServer def ehCacheManager() = ehCacheManagerFactoryBean.getObject @Bean def ehCacheManagerFactoryBean = { val ehCacheManagerFactoryBean = new EhCacheManagerFactoryBean ehCacheManagerFactoryBean.setShared(true) ehCacheManagerFactoryBean.setCacheManagerName("spring-pitfalls") ehCacheManagerFactoryBean } Note that I explicitly set CacheManager name. This is not required but this name is used as part of the Mbean name and a default one contains hashCode value, which is not very pleasant. The final touch is to enable statistics on a cache basis: Now we can happily monitor various caching characteristics of every cache separately: As we can see the percentage of cache misses increases. Never a good thing. If we don't enable cache statistics, enabling JMX is still a good idea since we get a lot of management operations for free, including flushing and clearing caches (useful during debugging and testing). Quartz scheduler In my humble opinion Quartz scheduler is very underestimated library, but I will write an article about it on its own. This time we will only learn how to monitor it via JMX. Fortunately it's as simple as adding: org.quartz.scheduler.jmx.export=true To quartz.properties file. The JMX support in Quartz could have been slightly broader, but still one can query e.g. which jobs are currently running. By the way the new major version of Quartz (2.x) brings very nice DSL-like support for scheduling: val job = newJob(classOf[MyJob]) val trigger = newTrigger(). withSchedule( repeatSecondlyForever() ). startAt( futureDate(30, SECOND) ) scheduler.scheduleJob(job.build(), trigger.build()) Apache Commons DBCP Apache Commons DBCP is the most reasonable JDBC pooling library I came across. There is also c3p0, but it doesn't seem like it's actively developed any more. Tomcat JDBC Connection Pool looked promising, but since it's bundled in Tomcat, your JDBC drivers can no longer be packaged in WAR. The only problem with DBCP is that it does not support JMX. At all (see this two and a half year old issue). Fortunately this can be easily worked around. Besides we will learn how to use Spring built-in JMX support. Looks like the standard BasicDataSource has all what we need, all we have to do is to expose existing metrics via JMX. With Spring it is dead-simple – just subclass BasicDataSource and add @ManagedAttribute annotation over desired attributes: @ManagedResource class ManagedBasicDataSource extends BasicDataSource { @ManagedAttribute override def getNumActive = super.getNumActive @ManagedAttribute override def getNumIdle = super.getNumIdle @ManagedAttribute def getNumOpen = getNumActive + getNumIdle @ManagedAttribute override def getMaxActive: Int= super.getMaxActive @ManagedAttribute override def setMaxActive(maxActive: Int) { super.setMaxActive(maxActive) } @ManagedAttribute override def getMaxIdle = super.getMaxIdle @ManagedAttribute override def setMaxIdle(maxIdle: Int) { super.setMaxIdle(maxIdle) } @ManagedAttribute override def getMinIdle = super.getMinIdle @ManagedAttribute override def setMinIdle(minIdle: Int) { super.setMinIdle(minIdle) } @ManagedAttribute override def getMaxWait = super.getMaxWait @ManagedAttribute override def setMaxWait(maxWait: Long) { super.setMaxWait(maxWait) } @ManagedAttribute override def getUrl = super.getUrl @ManagedAttribute override def getUsername = super.getUsername } Here are few data source metrics going crazy during load-test: JMX support in the Spring framework itself is pretty simple. As you have seen above exposing arbitrary attribute or operation is just a matter of adding an annotation. You only have to remember about enabling JMX support using either XML or Java (also see: SPR-8943 : Annotation equivalent to with @Configuration): or: @Bean def annotationMBeanExporter() = new AnnotationMBeanExporter() This article wasn't particularly exciting. However, the knowledge of JMX metrics will enable us to write simple yet fancy dashboards in no time. Stay tuned! From http://nurkiewicz.blogspot.com/2011/12/enabling-jmx-in-hibernate-ehcache-qurtz.html
December 22, 2011
by Tomasz Nurkiewicz
· 12,874 Views
article thumbnail
How to create offline HTML5 web apps in 5 easy steps
Among all cool new features introduced by HTML5, the possibility of caching web pages for offline use is definitely one of my favorites. Today, I’m glad to show you how you can create a page that will be available for offline browsing. Getting started View Demo Download files 1 – Add HTML5 doctype The first thing to do is create a valid HTML5 document. The HTML5 doctype is easier to remember than ones used for xhtml: ... Create a file named index.html, or get the example files from my CSS3 media queries article to use as a basis for this tutorial. In case you need it, the full HTML5 specs are available on the W3C website. 2 – Add .htaccess support The file we’re going to create to cache our web page is called a manifest file. Before creating it, we first have to add a directive to the .htaccess file (assuming your server is Apache). Open the .htaccess file, which is located on your website root, and add the following code: AddType text/cache-manifest .manifest This directive makes sure that every .manifest file is served as text/cache-manifest. If the file isn’t, then the whole manifest will have no effect and the page will not be available offline. 3 – Create the manifest file Now, things are going to be more interesting as we create a manifest file. Create a new file and save it as offline.manifest. Then, paste the following code in it. I’ll explain it later. CACHE MANIFEST #This is a comment CACHE index.html style.css image.jpg image-med.jpg image-small.jpg notre-dame.jpg Right now, you have a perfectly working manifest file. The way it works is very simple: After the CACHE declaration, you have to list each files you want to make available offline. That’s enough for caching a simple web page like the one from my example, but HTML5 caching has other interesting possibilities. For example, consider the following manifest file: CACHE MANIFEST #This is a comment CACHE index.html style.css NETWORK: search.php login.php FALLBACK: /api offline.html Like in the example manifest file, we have a CACHE declaration that caches index.html and style.css. But we also have the NETWORK declaration, which is used to specify files that shouldn’t be cached, such as a login page. The last declaration is FALLBACK. This declaration allows you to redirect the user to a particular file (in this example, offline.html) if a resource (/api) isn’t available offline. 4 – Link your manifest file to the html document Now, both your manifest file and your main html document are ready. The only thing you still have to do is to link the manifest file to the html document. Doing this is easy: simply add the manifest attribute to the html element as shown below: 5 – Test it Once done, you’re ready to go. If you visit your index.html file with Firefox 3.5+, you should see a banner like this one: Other browser I’ve tested (Chrome, Safari, Android and iPhone) do not warn about the file caching, and the file is automatically cached. Below you’ll find the browser compatibility of this technique: As usual Internet Explorer does not support it. IE: No support Firefox: 3.5+ Safari: 4.0+ Chrome: 5.0+ Opera: 10.6+ iPhone: 2.1+ Android: 2.0+ Source: http://www.catswhocode.com/blog/how-to-create-offline-html5-web-apps-in-5-easy-steps
December 22, 2011
by Jean-Baptiste Jung
· 24,316 Views
article thumbnail
A Gentle Introduction to Making HTML5 Canvas Interactive
this is an big overhaul of one of my tutorials on making and moving shapes on an html5 canvas. this new tutorial is vastly cleaner than my old one, but if you still want to see that one or are looking for the concept of a “ghost context” you can find that one here. this tutorial will show you how to create a simple data structure for shapes on an html5 canvas and how to have them be selectable. the finished canvas will look like this: this article’s code is written primarily to be easy to understand. we’ll be going over a few things that are essential to interactive apps such as games (drawing loop, hit testing), and in later tutorials i will probably turn this example into a small game of some kind. i will try to accommodate javascript beginners but this introduction does expect at least a rudimentary understanding of js. not every piece of code is explained in the text, but almost every piece of code is thoroughly commented! the html5 canvas a canvas is made by using the tag in html: this text is displayed if your browser does not support html5 canvas. a canvas isn’t smart: it’s just a place for drawing pixels. if you ask it to draw something it will execute the drawing command and then immediately forget everything about what it has just drawn. this is sometimes referred to as an immediate drawing surface, as contrasted with svg as a retained drawing surface, since svg keeps a reference to everything drawn. because we have no such references, we have to keep track ourselves of all the things we want to draw (and re-draw) each frame. canvas also has no built-in way of dealing with animation. if you want to make something you’ve drawn move, you have to clear the entire canvas and redraw all of the objects with one or more of them moved. and you have to do it often, of course, if you want a semblance of animation or motion. so we’ll need to add: code for keeping track of objects code for keeping track of canvas state code for mouse events code for drawing the objects as they are made and move around keeping track of what we draw to keep things simple for this example we will start with a shape class to represent rectangular objects. javascript doesn’t technically have classes, but that isn’t a problem because javascript programmers are very good at playing pretend. functionally (well, for our example) we are going to have a shape class and create shape instances with it. what we are really doing is defining a function named shape and adding functions to shape’s prototype. you can make new instances of the function shape and all instances will share the functions defined on shape’s prototype. if you’ve never encountered prototypes in javascript before or if the above sounds confusing to you, i highly recommend reading crockford’s javascript: the good parts . the book is an intermediate overview of javascript that gives a good understanding of why programmers choose to create objects in different ways, why certain conventions are frowned upon, and just what makes javascript so different. here’s our shape constructor and one of the two prototype methods, which are comparable to a class instance methods: // constructor for shape objects to hold data for all drawn objects. // for now they will just be defined as rectangles. function shape(x, y, w, h, fill) { // this is a very simple and unsafe constructor. // all we're doing is checking if the values exist. // "x || 0" just means "if there is a value for x, use that. otherwise use 0." this.x = x || 0; this.y = y || 0; this.w = w || 1; this.h = h || 1; this.fill = fill || '#aaaaaa'; } // draws this shape to a given context shape.prototype.draw = function(ctx) { ctx.fillstyle = this.fill; ctx.fillrect(this.x, this.y, this.w, this.h); } keeping track of canvas state we’re going to have a second class/function called canvasstate. we’re only going to make one instance of this class and it will hold all of the state in this tutorial that is not associated with shapes themselves. canvasstate is going to foremost need a reference to the canvas and a few other field for convenience. we’re also going to compute and save the border and padding (if there is any) so that we can get accurate mouse coordinates. in the canvasstate constructor we will also have a collection of state relating to the objects on the canvas and the current status of dragging. we’ll make an array of shapes, a flag “dragging” that will be true while we are dragging, a field to keep track of which object is selected and a “valid” flag that will be set to false will cause the canvas to clear everything and redraw. is going to need an array of shapes to keep track of whats been drawn so far. i’m going to add a bunch of variables for keeping track of the drawing and mouse state. i already added boxes[] to keep track of each object, but we’ll also need a var for the canvas, the canvas’ 2d context (where wall drawing is done), whether the mouse is dragging, width/height of the canvas, and so on. we’ll also want to make a second canvas, for selection purposes, but i’ll talk about that later. function canvasstate(canvas) { // ... // i removed some setup code to save space // see the full source at the end // **** keep track of state! **** this.valid = false; // when set to true, the canvas will redraw everything this.shapes = []; // the collection of things to be drawn this.dragging = false; // keep track of when we are dragging // the current selected object. // in the future we could turn this into an array for multiple selection this.selection = null; this.dragoffx = 0; // see mousedown and mousemove events for explanation this.dragoffy = 0; mouse events we’ll add events for mousedown, mouseup, and mousemove that will control when an object starts and stops dragging. we’ll also disable the selectstart event, which stops double-clicking on canvas from accidentally selecting text on the page. finally we’ll add a double-click event that will create a new shape and add it to the canvasstate’s list of shapes. the mousedown event begins by calling getmouse on our canvasstate to return the x and y position of the mouse. we then iterate through the list of shapes to see if any of them contain the mouse position. we go through them backwards because they are drawn forwards, and we want to select the one that appears topmost, so we must find the potential shape that was drawn last. if we find the shape we save the offset, save that shape as our selection, set dragging to true and set the valid flag to false. already we’ve used most of our state! finally if we didn’t find any objects we need to see if there was a selection saved from last time. if there is we should clear it. since we clicked on nothing, we obviously didn’t click on the already-selected object! clearing the selection means we will have to clear the canvas and redraw everything without the selection ring, so we set the valid flag to false. // ... // (we are still in the canvasstate constructor) // this is an example of a closure! // right here "this" means the canvasstate. but we are making events on the canvas itself, // and when the events are fired on the canvas the variable "this" is going to mean the canvas! // since we still want to use this particular canvasstate in the events we have to save a reference to it. // this is our reference! var mystate = this; //fixes a problem where double clicking causes text to get selected on the canvas canvas.addeventlistener('selectstart', function(e) { e.preventdefault(); return false; }, false); // up, down, and move are for dragging canvas.addeventlistener('mousedown', function(e) { var mouse = mystate.getmouse(e); var mx = mouse.x; var my = mouse.y; var shapes = mystate.shapes; var l = shapes.length; for (var i = l-1; i >= 0; i--) { if (shapes[i].contains(mx, my)) { var mysel = shapes[i]; // keep track of where in the object we clicked // so we can move it smoothly (see mousemove) mystate.dragoffx = mx - mysel.x; mystate.dragoffy = my - mysel.y; mystate.dragging = true; mystate.selection = mysel; mystate.valid = false; return; } } // havent returned means we have failed to select anything. // if there was an object selected, we deselect it if (mystate.selection) { mystate.selection = null; mystate.valid = false; // need to clear the old selection border } }, true); the mousemove event checks to see if we have set the dragging flag to true. if we have it gets the current mouse positon and moves the selected object to that position, remembering the offset of where we were grabbing it. if the dragging flag is false the mousemove event does nothing. canvas.addeventlistener('mousemove', function(e) { if (mystate.dragging){ var mouse = mystate.getmouse(e); // we don't want to drag the object by its top-left corner, // we want to drag from where we clicked. // thats why we saved the offset and use it here mystate.selection.x = mouse.x - mystate.dragoffx; mystate.selection.y = mouse.y - mystate.dragoffy; mystate.valid = false; // something's dragging so we must redraw } }, true); the mouseup event is simple, all it has to do is update the canvasstate so that we are no longer dragging! so once you lift the mouse, the mousemove event is back to doing nothing. canvas.addeventlistener('mouseup', function(e) { mystate.dragging = false; }, true); the double click event we’ll use to add more shapes to our canvas. it calls addshape on the canvasstate with a new instance of shape. all addshape does is add the argument to the list of shapes in the canvasstate. // double click for making new shapes canvas.addeventlistener('dblclick', function(e) { var mouse = mystate.getmouse(e); mystate.addshape(new shape(mouse.x - 10, mouse.y - 10, 20, 20, 'rgba(0,255,0,.6)')); }, true); there are a few options i implemented, what the selection ring looks like and how often we redraw. setinterval simply calls our canvasstate’s draw method. our interval of 30 means that we call the draw method every 30 milliseconds. // **** options! **** this.selectioncolor = '#cc0000'; this.selectionwidth = 2; this.interval = 30; setinterval(function() { mystate.draw(); }, mystate.interval); } drawing now we’re set up to draw every 30 milliseconds, which will allow us to continuously update the canvas so it appears like the shapes we drag are smoothly moving around. however, drawing doesn’t just mean drawing the shapes over and over; we also have to clear the canvas on every draw. if we don’t clear it, dragging will look like the shape is making a solid line because none of the old shape-positions will go away. because of this, we clear the entire canvas before each draw frame. this can get expensive, and we only want to draw if something has actually changed within our framework, which is why we have the “valid” flag in our canvasstate. after everything is drawn the draw method will set the valid flag to true. then, ocne we do something like adding a new shape or trying to drag a shape, the state will get invalidated and draw() will clear, redraw all objects, and set the valid flag again. // while draw is called as often as the interval variable demands, // it only ever does something if the canvas gets invalidated by our code canvasstate.prototype.draw = function() { // if our state is invalid, redraw and validate! if (!this.valid) { var ctx = this.ctx; var shapes = this.shapes; this.clear(); // ** add stuff you want drawn in the background all the time here ** // draw all shapes var l = shapes.length; for (var i = 0; i < l; i++) { var shape = shapes[i]; // we can skip the drawing of elements that have moved off the screen: if (shape.x > this.width || shape.y > this.height || shape.x + shape.w < 0 || shape.y + shape.h < 0) return; shapes[i].draw(ctx); } // draw selection // right now this is just a stroke along the edge of the selected shape if (this.selection != null) { ctx.strokestyle = this.selectioncolor; ctx.linewidth = this.selectionwidth; var mysel = this.selection; ctx.strokerect(mysel.x,mysel.y,mysel.w,mysel.h); } // ** add stuff you want drawn on top all the time here ** this.valid = true; } } we go through all of shapes[] and draw each one in order. this will give the nice appearance of later shapes looking as if they are on top of earlier shapes. after all the shapes are drawn, a selection handle (if there is a selection) gets drawn around the shape that this.selection references. if you wanted a background (like a city) or a foreground (like clouds), one way to add them is to put them before or after the main two drawing bits. there are often better ways though, like using multiple canvases or a css background-image, but we won’t go over that here. getting mouse coordinates on canvas getting good mouse coordinates is a little tricky on canvas. you could use offsetx/y and layerx/y, but layerx/y is deprecated in webkit (chrome and safari) and firefox does not have offsetx/y. the most bulletproof way to get the correct mouse position is shown below. you have to walk up the tree adding the offsets together. then you must add any padding or border to the offset. finally, to fix coordinate problems when you have fixed-position elements on the page (like the wordpress admin bar or a stumbleupon bar) you must add the ’s offsettop and offsetleft. then you simply subtract that offset from the e.pagex/y values and you’ll get perfect coordinates in almost every possible situation. // creates an object with x and y defined, // set to the mouse position relative to the state's canvas // if you wanna be super-correct this can be tricky, // we have to worry about padding and borders canvasstate.prototype.getmouse = function(e) { var element = this.canvas, offsetx = 0, offsety = 0, mx, my; // compute the total offset if (element.offsetparent !== undefined) { do { offsetx += element.offsetleft; offsety += element.offsettop; } while ((element = element.offsetparent)); } // add padding and border style widths to offset // also add the offsets in case there's a position:fixed bar offsetx += this.stylepaddingleft + this.styleborderleft + this.htmlleft; offsety += this.stylepaddingtop + this.stylebordertop + this.htmltop; mx = e.pagex - offsetx; my = e.pagey - offsety; // we return a simple javascript object (a hash) with x and y defined return {x: mx, y: my}; } there are a few little methods i added that are not shown, such as shape’s method to see if a point is inside its bounds. you can see and download the full demo source here . now that we have a basic structure down, it is easy to write code that handles more complex shapes, like paths or images or video. rotation and scaling these things takes a bit more work, but is quite doable with the canvas and our selection method is already set up to deal with them. if you would like to see this code enhanced in future posts (or have any fixes), let me know. source: http://simonsarris.com/blog/510-making-html5-canvas-useful
December 21, 2011
by Simon Sarris
· 11,496 Views · 1 Like
article thumbnail
Google Guava Cache
This Post is a continuation of my series on Google Guava, this time covering Guava Cache. Guava Cache offers more flexibility and power than either a HashMap or ConcurrentHashMap, but is not as heavy as using EHCache or Memcached (or robust for that matter, as Guava Cache operates solely in memory). The Cache interface has methods you would expect to see like ‘get’, and ‘invalidate’. A method you won’t find is ‘put’, because Guava Cache is ‘self-populating’, values that aren’t present when requested are fetched or calculated, then stored. This means a ‘get’ call will never return null. In all fairness, the previous statement is not %100 accurate. There is another method ‘asMap’ that exposes the entries in the cache as a thread safe map. Using ‘asMap’ will result in not having any of the self loading operations performed, so calls to ‘get’ will return null if the value is not present (What fun is that?). Although this is a post about Guava Cache, I am going to spend the bulk of the time talking about CacheLoader and CacheBuilder. CacheLoader specifies how to load values, and CacheBuilder is used to set the desired features and actually build the cache. CacheLoader CacheLoader is an abstract class that specifies how to calculate or load values, if not present. There are two ways to create an instance of a CacheLoader: Extend the CacheLoader class Use the static factory method CacheLoader.from If you extend CacheLoader you need to override the V load(K key) method, instructing how to generate the value for a given key. Using the static CacheLoader.from method you build a CacheLoader either by supplying a Function or Supplier interface. When supplying a Function object, the Function is applied to the key to calculate or retrieve the results. Using a Supplier interface the value is obtained independent of the key. CacheBuilder The CacheBuilder is used to construct cache instances. It uses the fluent style of building and gives you the option of setting the following properties on the cache: Cache Size limit (removals use a LRU algorithm) Wrapping keys in WeakReferences (Strong references used by default for keys) Wrapping values in either WeakReferences or SoftReferences (Strong references used by default) Time to expire entires after last access Time based expiration of entries after being written or updated Setting a RemovalListener that can recieve events once an entry is removed from the cache Concurrency Level of the cache (defaults to 4) The concurrency level option is used to partition the table internally such that updates can occur without contention. The ideal setting would be the maximum number of threads that could potentially access the cache at one time. Here is an example of a possible usage scenario for Guava Cache. public class PersonSearchServiceImpl implements SearchService> { public PersonSearchServiceImpl(SampleLuceneSearcher luceneSearcher, SampleDBService dbService) { this.luceneSearcher = luceneSearcher; this.dbService = dbService; buildCache(); } @Override public List search(String query) throws Exception { return cache.get(query); } private void buildCache() { cache = CacheBuilder.newBuilder().expireAfterWrite(10, TimeUnit.MINUTES) .maximumSize(1000) .build(new CacheLoader>() { @Override public List load(String queryKey) throws Exception { List ids = luceneSearcher.search(queryKey); return dbService.getPersonsById(ids); } }); } } In this example, I am setting the cache entries to expire after 10 minutes of being written or updated in the cache, with a maximum amount of 1,000 entires. Note the usage of CacheLoader on line 15. RemovalListener The RemovalListener will receive notification of an item being removed from the cache. These notifications could be from manual invalidations or from a automatic one due to time expiration or garbage collection. The RemovalListener parameters can be set to listen for specific type. To receive notifications for any key or value set them to use Object. It should be noted here that a RemovalListener will receive a RemovalNotification object that implements the Map.Entry interface. The key or value could be null if either has already been garbage collected. Also the key and value object will be strong references, regardless of the type of references used by the cache. CacheStats There is also a very useful class CacheStats that can be retrieved via a call to Cache.stats(). The CacheStats object can give insight into the effectiveness and performance of your cache by providing statistics such as: hit count miss count total load timme total requests CacheStats provides many other counts in addition to the ones listed above. Conclusion The Guava Cache presents some very compelling functionality. The decision to use a Guava Cache really comes down to the tradeoff between memory availability/usage versus increases in performance. I have added a unit test CacheTest demonstrating the usages discussed here. As alway comments and suggestions are welcomed. Thanks for your time. Resources Guava Project Home Cache API Source Code for blog series From http://codingjunkie.net/google-guava-cache/
December 16, 2011
by Bill Bejeck
· 53,107 Views · 1 Like
article thumbnail
How To Sort String Array Using LINQ In C#
A string[] array and use a LINQ query expression to order its contents alphabetically. Note that we are ordering the strings, not the letters in the strings. using System; using System.Linq; class Program { static void Main() { string[] a = new string[] {"Indonesian","Korean","Japanese","English","German"}; var sort = from s in a orderby s select s; foreach (string c in sort) { Console.WriteLine(c); } } } /*OUTPUT English German Indonesian Japanese Korean */ Eclipse IDE and Eclipse
December 14, 2011
by Snippets Manager
· 11,003 Views · 1 Like
article thumbnail
Easy URL rewriting in ASP.NET 4.0 web forms
In this post I am going to explain URL rewriting in greater details. This post will contain basic of URL rewriting and will explain how we can do URL rewriting in fewer lines of code. Why we need URL rewriting? Let’s consider a simpler scenario we want to display a customer details on a ASP.NET Page so how our page will know that for which customer we need to display details? The simplest way of doing is to use query string we will pass a customer id which uniquely identifies customer in query string. So our url will look like this. Customer.aspx?Id=1 This will work but the problem with above URL is that its not user friendly and search engine friendly. who is going to remember that what query string parameter I am going to pass and why we need that parameter. Also when search engine will crawl this site it will going to read this URL blindly as this url is not informative because it query string is not readable for search engine crawlers. So your search engine will be ranked lower as this URL is not readable to search engine crawlers.Now when do a URL rewriting our URL will be cleaner shorter and simpler like this. Customers/Id/1/ Here anybody in world can understand it talking about customer and this page will used to show customer details.Even search engine crawler will also know that you are talking about customers. That is why we need URL rewriting. URL rewriting and ASP.NET In earlier versions of ASP.NET we have to write lots of code for URL rewriting but Now with ASP.NET 4.0 you can easily rewrite in fewer lines of code. So let’s start a Demo where I will demonstrate you how we can easily rewrite URLs. So let’s first create a ASP. NET web form application via File->New->Project and a dialog box will open just like below and then created a empty project called URL rewriting. After creating a project I have added global.asax – where we are going to write url mapping logic and then I have added an asp.net page called which will display customer information just like below. So everything is now ready let’s start writing code. First thing we need to do is to define routes. Route will map a URL to physical page.First I will create static function called Register route which map route to particular file and then I am going to call this from application_start event of global.asax. Following is code for that. using System; using System.Web.Routing; namespace UrlRewriting { public class Global : System.Web.HttpApplication { protected void Application_Start(object sender, EventArgs e) { RegisterRoutes(RouteTable.Routes); } public static void RegisterRoutes(RouteCollection routeCollection) { routeCollection.MapPageRoute("RouteForCustomer", "Customer/{Id}", "~/Customer.aspx"); } } } Now as mapping code has been done let’s right code for customer.aspx page. I have have following code in page_load event of customer.aspx using System; namespace UrlRewriting { public partial class Customer : System.Web.UI.Page { protected void Page_Load(object sender, EventArgs e) { string id = Page.RouteData.Values["Id"].ToString(); Response.Write("Customer Details page"); Response.Write(string.Format("Displaying information for customer : {0}",id)); } } } Here in above code you can see that I am getting value from page route data and then just printing it. In real world it will fetch customer data from database and show customer details on page. Now let’s run that application. It will print a details of customer as I have passed Id in URL suppose you pass 1 as id in URL then it will look like following. Now if you put 2 in url it will print information about customer 2. That’s it. So we have enabled URL rewriting in asp.net in fewer lines of code. In next post I am going to explain redirection with URL rewriting. Hope you like this post. Stay tuned for more.. Till then Happy programming
December 10, 2011
by Jalpesh Vadgama
· 28,700 Views
article thumbnail
MySQL vs. Neo4j on a Large-Scale Graph Traversal
this post presents an analysis of mysql (a relational database) and neo4j (a graph database) in a side-by-side comparison on a simple graph traversal. the data set that was used was an artificially generated graph with natural statistics. the graph has 1 million vertices and 4 million edges. the degree distribution of this graph on a log-log plot is provided below. a visualization of a 1,000 vertex subset of the graph is diagrammed above. loading the graph the graph data set was loaded both into mysql and neo4j. in mysql a single table was used with the following schema. create table graph ( outv int not null, inv int not null ); create index outv_index using btree on graph (outv); create index inv_index using btree on graph (inv); after loading the data, the table appears as below. the first line reads: “vertex 0 is connected to vertex 1.” mysql> select * from graph limit 10; +------+-----+ | outv | inv | +------+-----+ | 0 | 1 | | 0 | 2 | | 0 | 6 | | 0 | 7 | | 0 | 8 | | 0 | 9 | | 0 | 10 | | 0 | 12 | | 0 | 19 | | 0 | 25 | +------+-----+ 10 rows in set (0.04 sec) the 1 million vertex graph data set was also loaded into neo4j. in gremlin , the graph edges appear as below. the first line reads: “vertex 0 is connected to vertex 992915.” gremlin> g.e[1..10] ==>e[183][0-related->992915] ==>e[182][0-related->952836] ==>e[181][0-related->910150] ==>e[180][0-related->897901] ==>e[179][0-related->871349] ==>e[178][0-related->857804] ==>e[177][0-related->798969] ==>e[176][0-related->773168] ==>e[175][0-related->725516] ==>e[174][0-related->700292] warming up the caches before traversing the graph data structure in both mysql and neo4j, each database had a “ warm up ” procedure run on it. in mysql, a “select * from graph” was evaluated and all of the results were iterated through. in neo4j, every vertex in the graph was iterated through and the outgoing edges of each vertex were retrieved. finally, for both mysql and neo4j, the experiment discussed next was run twice in a row and the results of the second run were evaluated. traversing the graph the traversal that was evaluated on each database started from some root vertex and emanated n-steps out. there was no sorting, no distinct-ing, etc. the only two variables for the experiments are the length of the traversal and the root vertex to start the traversal from. in mysql, the following 5 queries denote traversals of length 1 through 5. note that the “?” is a variable parameter of the query that denotes the root vertex. select a.inv from graph as a where a.outv=? select b.inv from graph as a, graph as b where a.inv=b.outv and a.outv=? select c.inv from graph as a, graph as b, graph as c where a.inv=b.outv and b.inv=c.outv and a.outv=? select d.inv from graph as a, graph as b, graph as c, graph as d where a.inv=b.outv and b.inv=c.outv and c.inv=d.outv and a.outv=? select e.inv from graph as a, graph as b, graph as c, graph as d, graph as e where a.inv=b.outv and b.inv=c.outv and c.inv=d.outv and d.inv=e.outv and a.outv=? for neo4j, the blueprints pipes framework was used. a pipe of length n was constructed using the following static method. public static pipeline createpipeline(final integer steps) { final arraylist pipes = new arraylist(); for (int i = 0; i < steps; i++) { pipe pipe1 = new vertexedgepipe(vertexedgepipe.step.out_edges); pipe pipe2 = new edgevertexpipe(edgevertexpipe.step.in_vertex); pipes.add(pipe1); pipes.add(pipe2); } return new pipeline(pipes); } for both mysql and neo4j, the results of the query (sql and pipes) were iterated through. thus, all results were retrieved for each query. in mysql, this was done as follows. while (resultset.next()) { resultset.getint(finalcolumn); } in neo4j, this is done as follows. while (pipeline.hasnext()) { pipeline.next(); } experimental results the artificial graph dataset was constructed with a “ rich get richer “, preferential attachment model . thus, the vertices created earlier are the most dense (i.e. highest number of adjacent vertices). this property was used to limit the amount of time it would take to evaluate the tests for each traversal. only the first 250 vertices were used as roots of the traversals. before presenting timing results, note that all of these experiments were run on a macbook pro with a 2.66ghz intel core 2 duo and 4gigs of ram at 1067 mhz ddr3. the packages used were java 1.6, mysql jdbc 5.0.8, and blueprints pipes 0.1.2. java version "1.6.0_17" java(tm) se runtime environment (build 1.6.0_17-b04-248-10m3025) java hotspot(tm) 64-bit server vm (build 14.3-b01-101, mixed mode) the following java virtual machine parameters were used: -xmx1000m -xms500m below are the total running times for both mysql (red) and neo4j (blue) for traversals of length 1, 2, 3, and 4. the raw data is presented below along with the total number of vertices returned by each traversal—which, of course, is the same for both mysql and neo4j given that its the same graph data set being processed. also realize that traversals can loop and thus, many of the same vertices are returned multiple times. finally, note that only neo4j has the running time for a traversal of length 5. mysql did not finish after waiting 2 hours to complete. in comparison, neo4j took 14.37 minutes to complete a 5 step traversal. [mysql steps-1] time(ms):124 -- vertices_returned:11360 [mysql steps-2] time(ms):922 -- vertices_returned:162640 [mysql steps-3] time(ms):8851 -- vertices_returned:2206437 [mysql steps-4] time(ms):112930 -- vertices_returned:28125623 [mysql steps-5] n/a [neo4j steps-1] time(ms):27 -- vertices_returned:11360 [neo4j steps-2] time(ms):474 -- vertices_returned:162640 [neo4j steps-3] time(ms):3366 -- vertices_returned:2206437 [neo4j steps-4] time(ms):49312 -- vertices_returned:28125623 [neo4j steps-5] time(ms):862399 -- vertices_returned:358765631 next, the individual data points for both mysql and neo4j are presented in the plot below. each point denotes how long it took to return n number of vertices for the varying traversal lengths. finally, the data below provides the number of vertices returned per millisecond (on average) for each of the traversals. again, mysql did not finish in its 2 hour limit for a traversal of length 5. [mysql steps-1] vertices/ms:91.6128847554668 [mysql steps-2] vertices/ms:176.399127537985 [mysql steps-3] vertices/ms:249.286746556076 [mysql steps-4] vertices/ms:249.053599519823 [mysql steps-5] n/a [neo4j steps-1] vertices/ms:420.740351166341 [neo4j steps-2] vertices/ms:343.122344772028 [neo4j steps-3] vertices/ms:655.507125256186 [neo4j steps-4] vertices/ms:570.360621871775 [neo4j steps-5] vertices/ms:416.00886711325 conclusion in conclusion, given a traversal of an artificial graph with natural statistics, the graph database neo4j is more optimal than the relational database mysql. however, no attempts have been made to optimize the java vm, the sql queries, etc. these experiments were run with both neo4j and mysql “out of the box” and with a “natural syntax” for both types of queries. source: http://markorodriguez.com/2011/02/18/mysql-vs-neo4j-on-a-large-scale-graph-traversal/
December 5, 2011
by Marko Rodriguez
· 58,747 Views · 1 Like
article thumbnail
Create Your Own XML/JSON/HTML API with PHP
Develop your own API service for your PHP projects.
December 1, 2011
by Andrei Prikaznov
· 68,450 Views
article thumbnail
Zero Downtime – What is it and why is it important?
For most large web applications, uptime is of foremost importants. Any outage can be seen by customers as a frustration, or opportunity to move to a competitor. What's more for a site that also includes e-commerce, it can mean real lost sales. Zero Downtime describes a site without service interruption. To achieve such lofty goals, redundancy becomes a critical requirement at every level of your infrastructure. If you're using cloud hosting, are you redundant to alternate availability zones and regions? Are you using geographically distributed load balancing? Do you have multiple clustered databases on the backend, and multiple webservers load balanced. All of these requirements will increase uptime, but may not bring you close to zero downtime. For that you'll need thorough testing. The solution is to pull the trigger on sections of your infrastructure, and prove that it fails over quickly without noticeable outage. The ultimate test is the outage itself. Sean Hull on Quora: What is zero downtime and why is it important? Source: http://www.iheavy.com/2011/06/23/zero-downtime-what-is-it-and-why-is-it-important/
November 23, 2011
by Sean Hull
· 26,149 Views
article thumbnail
Eventual Consistency in NoSQL Databases: Theory and Practice
One of NoSQL's goals: handle previously-unthinkable amounts of data. One of unthinkable-amounts-of-data's problems: previously-improbable events become extremely probable, precisely because the set of interactions is so large. Flip a coin a hundred times, and you're not likely to get 50 heads in a row. But flip it a few trillion times, and you probably will find some 50-heads streaks. So NoSQL's performance strength is also its mathematical weakness. This order of scale can result in lots of problems, but one of the most common is consistency -- the C in ACID -- clearly a fundamental desideratum for any database system, but in principle much harder to acheive for NoSQL databases than for others. Emerging database technologies have forced developers and computer scientists to define more exactly what kind of consistency is really needed, for any given application. Two years ago, ACM (the Association for Computing Machinery) published an extremely helpful examination of the attenuated notion of consistency called 'eventual consistency'. Their summary: Data inconsistency in large-scale reliable distributed systems must be tolerated for two reasons: improving read and write performance under highly concurrent conditions; and handling partition cases where a majority model would render part of the system unavailable even though the nodes are up and running. The article surveys technical solutions as well as user considerations that might soften the undesirability of anything less than perfect, instantaneous consistency. It's not long (4 pages plus pictures), and explains some deep database issues quite clearly. On the more practical side of the problem: Russell Brown recently gave a talk at the NoSQL Exchange 2011 on exactly this topic. More specifically, he showed how some distributed systems (Riak in particular) try to minimize conflicts, and suggested some ways to reconcile conflicts automatically using smart semantic techniques. Check out the NoSQL Exchange page for Russell's talk here, which includes an embedded video. But read the ACM article first for a broader overview, since Russell launches into technical details pretty quickly.
November 22, 2011
by John Esposito
· 12,783 Views
article thumbnail
Adventures in Archiving with MongoDB
Most databases have them: a small handful of big tables that grow at a substantially faster rate than any of their peers (think tweets, clicks, check-ins, etc). Over time, the sheer size of these tables cause queries against them to slow down, and increase the size and time taken for backups. In some cases, not all of this data needs to be “live” so that it can be randomly accessed forever, and can be archived once some criteria has been met. I like to think of this as Eventually Irrelevant (if I may coin a term). This post was authored by Brian Ploetz Many database products support partitioning features, which allow you to arrange data physically based on some criteria (typically date, range, list, or hash-based partitioning). These partitioning features allow you to drop partitions from the live database when appropriate. While MongoDB supports horizontal partitioning via sharding, it does not currently support the notion of dropping shards. Even if it did, the criteria for what documents to drop from a collection would most likely not be based on your shard key, so it’s unlikely you’d want to drop a shard anyways. You’d want to drop data from all shards. Capped Collections and the forthcoming TTL-based Capped Collections feature are not quite what we’re looking for either. Capped Collections have very strict rules (including not being able to shard them), and for both normal and TTL-based Capped Collections, as old documents are aged out they are simply dropped on the floor. What we need is more control over the dropping process, such that we can guarantee that our data has been copied to an archive database/data warehouse before being dropped from the main database. I set out to find the most efficient way of manually archiving documents from a large collection given the tools currently available in MongoDB. For the purposes of this exercise, let’s assume we have a large collection of ad clicks which are associated with ads. We want to archive clicks which are associated with ads which expired more than 3 months ago. MapReduce MongoDB wants you to do bulk operations using MapReduce. Since all of the examples and documentation revolve around aggregation, I suspected I wouldn’t be able to use MapReduce as the backbone of my archiving process. After some experimenting, I found that my intuition was correct. Upon attempting to insert/remove documents within the map or reduce functions, I would quickly run into exception 10293 internal error: locks are not upgradeable errors: Sun Sep 25 21:22:24 [conn5] update warehouse.clicks query: { _id: ObjectId('4d6bf191db0b4b2b1fad5b65') } exception 10293 internal error: locks are not upgradeable: { "opid" : 4248173, "active" : false, "waitingForLock" : false, "op" : "update", "ns" : "?", "query" : { "_id" : { "$oid" : "4d6bf191db0b4b2b1fad5b65" } }, "client" : "0.0.0.0:0", "desc" : "conn" } 0ms Sun Sep 25 21:22:24 [conn5] Assertion: 10293:internal error: locks are not upgradeable: { "opid" : 4248174, "active" : false, "waitingForLock" : false, "op" : "update", "ns" : "?", "query" : { "_id" : { "$oid" : "4d6bf198db0b4b2b20ad5b65" } }, "client" : "0.0.0.0:0", "desc" : "conn" } 0x10008de9b 0x1002064f7 0x1002c1ec6 0x1002c4f3d 0x1002c5f7a 0x1000a8181 0x100147df4 0x1004dc68e 0x1004efc93 0x1004dc71b 0x1004dcb71 0x10049a078 0x100158efa 0x100377760 0x100389d0c 0x10034c204 0x10034d877 0x100180cc4 0x100184649 0x1002b9e89 0 mongod 0x000000010008de9b _ZN5mongo11msgassertedEiPKc + 315 1 mongod 0x00000001002064f7 _ZN5mongo10MongoMutex19_writeLockedAlreadyEv + 263 2 mongod 0x00000001002c1ec6 _ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE + 886 3 mongod 0x00000001002c4f3d _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE + 5661 4 mongod 0x00000001002c5f7a _ZN5mongo14DBDirectClient3sayERNS_7MessageE + 106 5 mongod 0x00000001000a8181 _ZN5mongo12DBClientBase6updateERKSsNS_5QueryENS_7BSONObjEbb + 273 6 mongod 0x0000000100147df4 _ZN5mongo12mongo_updateEP9JSContextP8JSObjectjPlS4_ + 660 7 mongod 0x00000001004dc68e js_Invoke + 3864 8 mongod 0x00000001004efc93 js_Interpret + 71932 9 mongod 0x00000001004dc71b js_Invoke + 4005 10 mongod 0x00000001004dcb71 js_InternalInvoke + 404 11 mongod 0x000000010049a078 JS_CallFunction + 86 12 mongod 0x0000000100158efa _ZN5mongo7SMScope6invokeEyRKNS_7BSONObjEib + 666 13 mongod 0x0000000100377760 _ZN5mongo2mr8JSMapper3mapERKNS_7BSONObjE + 96 14 mongod 0x0000000100389d0c _ZN5mongo2mr16MapReduceCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb + 1740 15 mongod 0x000000010034c204 _ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 628 16 mongod 0x000000010034d877 _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi + 2151 17 mongod 0x0000000100180cc4 _ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi + 52 18 mongod 0x0000000100184649 _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 10585 19 mongod 0x00000001002b9e89 _ZN5mongo13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 569 Upon killing the client which kicked off the MapReduce process, theses errors would continue to spin out of control, and I’d have to bounce the server. While MapReduce’s divide and conquer approach is ideal for what we’re trying to accomplish, unless the ability to obtain a write lock within a MapReduce function is supported, it is not a viable option for now. db.eval() While a server side script is certainly a viable option for writing an archiving process, it has a fundamental flaw in that it won’t work for sharded collections, which could be a non-starter for some folks. An example archiving process using db.eval() would look something like this: archiveClick = function archiveClick(doc) { if (expiredAdIds.indexOf(doc.ad_id) != -1) { archivedb = db.getMongo().getDB("archive"); archivedb.clicks.save(doc); // before we remove the original document, make sure the archive worked if (archivedb.getLastError() == null) { db.clicks.remove({_id: doc._id}); } else { throw "could not archive document with _id " + doc._id; } } } archiveClicks = function archiveClicks() { threeMonthsAgo = new Date(); threeMonthsAgo.setMonth(threeMonthsAgo.getMonth()-3); expiredAdIds = []; getExpiredAds = function(ad) {expiredAdIds.push(ad._id.str);} db.ads.find({end_date: {$lte: threeMonthsAgo}).forEach(getExpiredAds); db.system.js.save({_id: "expiredAdIds", value: expiredAdIds}); db.clicks.find().forEach(archiveClick); db.system.js.remove({_id: "expiredAdIds"}); } db.system.js.save({_id: "archiveClick", value: archiveClick}); db.system.js.save({_id: "archiveClicks", value: archiveClicks}); db.runCommand({$eval: "archiveClicks()", nolock: true}); Note the nolock: true sent to db.eval(). This ensures the mongod isn’t locked for other operations while this runs. The Holy Grail: Define Custom archive() Functions On Collections In order for our archiving process to work for sharded collections, we will need to add new functions to collection objects in the shell to perform the archiving. Frankly, this feels a lot more natural than db.eval(). I wanted to support two modes of archiving: immediate archiving, and a mark & sweep approach which allows you to queue documents to be archived at one point in time, and perform the actual archiving at a later time (off-peak hours). I’ve made these functions available in my mongodb-archive project on GitHub. Download the archive.js file, and use it like so: load("archive.js"); threeMonthsAgo = new Date(); threeMonthsAgo.setMonth(threeMonthsAgo.getMonth()-3); expiredAdIds = []; getExpiredAds = function(ad) {expiredAdIds.push(ad._id.str);} db.ads.find({end_date: {$lte: threeMonthsAgo}).forEach(getExpiredAds); archiveConnection = new Mongo("localhost:27018"); archiveDB = archiveConnection.getDB("archive"); archiveCollection = archiveDB.getCollection("clicks"); for (var i = 0; i < expiredAdIds.length; i++) { print("archiving clicks for ad_id: " + expiredAdIds[i]); db.clicks.archive({"ad_id": expiredAdIds[i]}, archiveCollection); print(""); } The mark/sweep variant would look like: db.clicks.queueForArchive({"ad_id": expiredAdIds[i]}); // ....some time later..... db.clicks.archiveQueued(archiveCollection); The biggest factors on the performance of the archiving process are really no different than the things that impact the performance of your MongoDB database overall: working set size, indexes, disk speed, and lock contention. The more indexes there are on the source collection, the longer the remove() will take. The more indexes there are on the archive collection, the longer the save() will take. The more of the data that you wish to archive is resident in memory, the faster the process will be. On my MacBook Pro (2.3GHz i7, 8GB RAM) with a 5400 RPM SATA drive and a clicks collection containing over 11 million documents, I was able to archive ~500 documents per second with the immediate archiving approach. Example output from mongostat when this was running: Archive DB: insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 1 0 0 3 0 208m 2.85g 15m 3.8 0 0|0 0|0 1k 1k 2 17:13:02 0 0 45 0 0 46 0 208m 2.85g 15m 3.4 0 0|0 0|0 61k 6k 2 17:13:03 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 8 0 0 9 0 208m 2.85g 15m 0.1 0 0|0 0|0 10k 2k 2 17:13:04 0 0 47 0 0 48 0 208m 2.85g 15m 3.4 0 0|0 0|0 66k 7k 2 17:13:05 0 0 0 0 0 1 1 208m 2.85g 15m 0 0 0|0 0|0 62b 1k 2 17:13:06 0 0 18 0 0 19 0 208m 2.85g 15m 2.6 0 0|0 0|0 24k 3k 2 17:13:07 0 0 79 0 0 80 0 208m 2.85g 16m 1.1 0 0|0 0|0 111k 11k 2 17:13:08 0 0 131 0 0 132 0 208m 2.85g 16m 8.8 0 0|0 0|0 181k 17k 2 17:13:09 0 0 216 0 0 217 0 208m 2.85g 16m 3.8 0 0|0 0|0 291k 27k 2 17:13:10 0 0 425 0 0 426 0 208m 2.85g 17m 10.5 0 0|0 0|0 600k 53k 2 17:13:11 0 0 490 0 0 490 0 208m 2.85g 19m 7.3 0 0|0 0|1 658k 61k 2 17:13:12 0 0 630 0 0 632 0 208m 2.85g 20m 11.3 0 0|0 0|0 844k 78k 2 17:13:13 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 818 0 0 818 0 208m 2.85g 22m 13.6 0 0|0 0|0 1m 101k 2 17:13:14 0 0 686 0 0 688 0 208m 2.85g 21m 9.6 0 0|0 0|0 932k 85k 2 17:13:15 0 0 0 0 0 1 0 208m 2.85g 22m 0 0 0|0 0|0 62b 1k 2 17:13:16 0 0 178 0 0 179 0 208m 2.85g 22m 1.2 0 0|0 0|0 252k 23k 2 17:13:17 0 0 974 0 0 975 0 208m 2.85g 23m 12.3 0 0|0 0|0 1m 121k 2 17:13:18 0 0 920 0 0 921 0 208m 2.85g 26m 10.1 0 0|0 0|0 1m 114k 2 17:13:19 0 0 810 0 0 811 0 208m 2.85g 26m 8.1 0 0|0 0|0 1m 100k 2 17:13:20 0 0 612 0 0 613 0 208m 2.85g 27m 8.1 0 0|0 0|0 798k 76k 2 17:13:21 0 0 0 0 0 1 0 208m 2.85g 27m 0 0 0|0 0|0 62b 1k 2 17:13:22 0 0 0 0 0 1 0 208m 2.85g 27m 0 0 0|0 0|0 62b 1k 2 17:13:23 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 30 0 0 31 0 208m 2.85g 27m 1.1 0 0|0 0|0 46k 5k 2 17:13:24 0 0 852 0 0 853 0 208m 2.85g 27m 14.8 0 0|0 0|0 1m 106k 2 17:13:25 0 0 768 0 0 769 0 208m 2.85g 29m 9.5 0 0|0 0|0 1m 95k 2 17:13:26 0 0 867 0 0 868 0 208m 2.85g 32m 13.4 0 0|0 0|0 1m 107k 2 17:13:27 0 0 705 0 0 706 0 208m 2.85g 31m 10.8 0 0|0 0|0 936k 88k 2 17:13:28 0 0 331 0 0 332 0 208m 2.85g 32m 3.9 0 0|0 0|0 446k 42k 2 17:13:29 0 0 0 0 0 1 0 208m 2.85g 32m 0 0 0|0 0|0 62b 1k 2 17:13:30 0 0 551 0 0 552 0 208m 2.85g 32m 6.1 0 0|0 0|0 739k 69k 2 17:13:31 0 0 728 0 0 729 0 208m 2.85g 35m 12.2 0 0|0 0|0 949k 90k 2 17:13:32 0 0 991 0 0 992 0 208m 2.85g 35m 11.7 0 0|0 0|0 1m 123k 2 17:13:33 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 935 0 0 935 0 208m 2.85g 38m 10.9 0 0|0 0|1 1m 116k 2 17:13:34 0 0 431 0 0 433 0 208m 2.85g 39m 5.9 0 0|0 0|0 563k 54k 2 17:13:35 0 0 0 0 0 1 0 208m 2.85g 39m 0 0 0|0 0|0 62b 1k 2 17:13:36 0 0 518 0 0 519 0 208m 2.85g 40m 5 0 0|0 0|0 693k 65k 2 17:13:37 0 0 904 0 0 905 0 208m 2.85g 40m 8.9 0 0|0 0|0 1m 112k 2 17:13:38 0 0 958 0 0 959 0 208m 2.85g 41m 14.2 0 0|0 0|0 1m 119k 2 17:13:39 0 0 899 0 0 900 0 208m 2.85g 44m 11.1 0 0|0 0|0 1m 111k 2 17:13:40 0 0 401 0 0 402 0 208m 2.85g 42m 5.9 0 0|0 0|0 540k 50k 2 17:13:41 0 0 856 0 0 857 0 208m 2.85g 44m 9.4 0 0|0 0|0 1m 106k 2 17:13:42 0 0 232 0 0 233 0 208m 2.85g 45m 2.9 0 0|0 0|0 311k 29k 1 17:13:43 Source DB: insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 2 0 12 1 13 0 56g 115g 2.82g 52.2 0 0|0 0|1 1k 822k 2 17:13:02 0 0 0 42 0 43 0 56g 115g 2.65g 88.1 0 0|0 0|1 5k 4k 2 17:13:03 0 0 0 14 0 15 1 56g 115g 1.88g 101 0 0|0 0|1 1k 2k 2 17:13:04 0 0 0 33 1 35 0 56g 115g 1.89g 53.8 16.6 0|0 1|0 4k 4k 2 17:13:05 0 0 0 0 0 1 0 56g 115g 1.9g 0 0 0|0 1|0 62b 1k 2 17:13:06 0 0 0 34 0 35 0 56g 115g 1.9g 47.3 0 0|0 0|0 4k 4m 2 17:13:07 0 0 0 91 0 92 0 56g 115g 1.89g 86.8 0 0|0 0|0 12k 8k 2 17:13:08 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 149 0 150 0 56g 115g 1.9g 82.5 0 0|0 0|0 20k 13k 2 17:13:09 0 0 0 287 0 287 0 56g 115g 1.9g 74.3 0 0|0 0|1 38k 25k 2 17:13:10 0 0 0 387 0 389 0 56g 115g 1.91g 64.6 0 0|0 0|0 52k 33k 2 17:13:11 0 0 0 580 0 581 0 56g 115g 1.93g 59 0 0|0 0|0 78k 49k 2 17:13:12 0 0 0 685 0 686 0 56g 115g 1.93g 47.9 0 0|0 0|0 93k 58k 2 17:13:13 0 0 0 729 0 730 0 56g 115g 1.95g 42.6 0 0|0 0|0 99k 61k 2 17:13:14 0 0 0 551 1 552 0 56g 115g 1.97g 34.5 0 0|0 1|0 74k 47k 2 17:13:15 0 0 0 0 0 1 0 56g 115g 1.98g 0 0 0|0 1|0 62b 1k 2 17:13:16 0 0 0 368 0 368 0 56g 115g 1.96g 20 0 0|0 0|1 50k 4m 2 17:13:17 0 0 0 1032 0 1034 0 56g 115g 2g 35.9 0 0|0 0|0 140k 87k 2 17:13:18 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 834 0 835 0 56g 115g 2.01g 42.3 0 0|0 0|0 113k 70k 2 17:13:19 0 0 0 922 0 922 0 56g 115g 2.02g 38.9 0 0|0 0|1 125k 77k 2 17:13:20 0 0 0 338 1 340 0 56g 115g 2.03g 13 0 0|0 1|0 46k 29k 2 17:13:21 0 0 0 0 0 1 0 56g 115g 2.04g 0 0 0|0 1|0 62b 1k 2 17:13:22 0 0 0 0 0 1 0 56g 115g 2.04g 0 0 0|0 1|0 62b 1k 2 17:13:23 0 0 0 185 0 186 0 56g 115g 2.02g 24.7 0 0|0 0|0 25k 4m 2 17:13:24 0 0 0 920 0 920 0 56g 115g 2.02g 29.9 0 0|0 0|1 125k 77k 2 17:13:25 0 0 0 741 0 742 0 56g 115g 2.04g 49 0 0|0 0|1 100k 62k 2 17:13:26 0 0 0 886 0 887 0 56g 115g 2.05g 33 0 0|0 0|1 120k 74k 2 17:13:27 0 0 0 683 0 684 0 56g 115g 2.08g 57.3 0 0|0 0|1 92k 58k 2 17:13:28 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 138 1 140 0 56g 115g 2.06g 14.5 0 0|0 1|0 18k 12k 2 17:13:29 0 0 0 12 0 12 0 56g 115g 2.07g 0.2 0 0|0 0|1 1k 4m 2 17:13:30 0 0 0 761 0 763 0 56g 115g 2.08g 46.5 0 0|0 0|0 103k 64k 2 17:13:31 0 0 0 762 0 762 0 56g 115g 2.09g 47 0 0|0 0|1 103k 64k 2 17:13:32 0 0 0 957 0 959 0 56g 115g 2.1g 35.3 0 0|0 0|0 130k 80k 2 17:13:33 0 0 0 905 0 905 0 56g 115g 2.13g 38.5 0 0|1 0|1 123k 76k 2 17:13:34 0 0 0 239 1 241 0 56g 115g 2.12g 14.9 0 0|0 1|0 32k 21k 2 17:13:35 0 0 0 0 0 1 0 56g 115g 2.13g 0 0 0|0 1|0 62b 1k 2 17:13:36 0 0 0 780 0 781 0 56g 115g 2.13g 42.8 0 0|0 0|0 106k 4m 2 17:13:37 0 0 0 805 0 806 0 56g 115g 2.14g 39.8 0 0|0 0|0 109k 68k 2 17:13:38 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 988 0 989 0 56g 115g 2.16g 34.8 0 0|0 0|0 134k 83k 2 17:13:39 0 0 0 953 0 953 0 56g 115g 2.17g 39.9 0 0|0 0|1 129k 80k 2 17:13:40 0 0 0 375 1 376 0 56g 115g 2.18g 13.9 0 0|1 0|1 51k 1m 2 17:13:41 0 0 0 867 0 869 0 56g 115g 2.19g 41.7 0 0|0 0|0 118k 73k 1 17:13:42 Total: MongoDB shell version: 2.0.1 connecting to: localhost:27017/prodcopy archiving clicks for ad_id: 4d6c12497b90420373000056 archiving documents... archived 19045 documents in 40701ms. For the queued approach with this same set up, I was able to queue ~1200 documents per second, and the archiving process performed at about the same ~500 documents per second. Example output from mongostat while this was running: Archive DB: insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:44 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:45 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:46 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:47 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:48 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:49 0 0 248 0 0 249 0 208m 2.85g 17m 6 0 0|0 0|0 283k 31k 2 17:16:50 0 0 780 0 0 781 0 208m 2.85g 18m 8.2 0 0|0 0|0 883k 97k 2 17:16:51 0 0 1256 0 0 1257 0 208m 2.85g 22m 13.7 0 0|0 0|0 1m 155k 2 17:16:52 0 0 1084 0 0 1085 0 208m 2.85g 21m 10 0 0|0 0|0 1m 134k 2 17:16:53 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 1108 0 0 1109 0 208m 2.85g 25m 9.8 0 0|0 0|0 1m 137k 2 17:16:54 0 0 1108 0 0 1109 0 208m 2.85g 24m 9.8 0 0|0 0|0 1m 137k 2 17:16:55 0 0 1110 0 0 1111 0 208m 2.85g 27m 11 0 0|0 0|0 1m 137k 2 17:16:56 0 0 1166 0 0 1167 0 208m 2.85g 31m 11.3 0 0|0 0|0 1m 144k 2 17:16:57 0 0 1246 0 0 1247 0 208m 2.85g 31m 10.6 0 0|0 0|0 1m 154k 2 17:16:58 0 0 1094 0 0 1095 0 208m 2.85g 35m 9.6 0 0|0 0|0 1m 135k 2 17:16:59 0 0 957 0 0 958 0 208m 2.85g 33m 9.6 0 0|0 0|0 1m 119k 2 17:17:00 0 0 721 0 0 722 0 464m 3.35g 38m 8.9 0 0|0 0|0 1m 90k 2 17:17:01 0 0 243 0 0 243 0 464m 3.35g 34m 9.7 0 0|0 0|0 445k 31k 2 17:17:02 0 0 921 0 0 923 0 464m 3.35g 37m 30.1 0 0|0 0|0 1m 114k 2 17:17:03 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 668 0 0 669 0 464m 3.35g 38m 9.6 0 0|0 0|0 962k 83k 2 17:17:04 0 0 995 0 0 995 0 464m 3.35g 41m 21.2 0 0|0 0|1 1m 123k 2 17:17:05 0 0 467 0 0 468 0 464m 3.35g 27m 12 0 0|1 0|1 679k 58k 2 17:17:06 0 0 1 0 0 2 1 464m 3.35g 17m 85.5 0 0|1 0|1 1k 1k 2 17:17:07 0 0 337 0 0 338 0 464m 3.35g 19m 36.2 0 0|0 0|1 499k 42k 2 17:17:08 0 0 917 0 0 919 0 464m 3.35g 21m 20.4 0 0|0 0|0 1m 114k 2 17:17:09 0 0 1022 0 0 1022 0 464m 3.35g 25m 22.7 0 0|0 0|1 1m 126k 2 17:17:10 0 0 970 0 0 971 0 464m 3.35g 24m 27.4 0 0|0 0|1 1m 120k 2 17:17:11 0 0 166 0 0 168 0 464m 3.35g 25m 1.2 0 0|0 0|0 251k 21k 2 17:17:12 0 0 862 0 0 863 0 464m 3.35g 28m 23.1 0 0|0 0|0 1m 107k 2 17:17:13 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 888 0 0 889 0 464m 3.35g 28m 28.8 0 0|0 0|0 1m 110k 2 17:17:14 0 0 540 0 0 541 0 464m 3.35g 30m 12.2 0 0|0 0|0 802k 67k 2 17:17:15 0 0 879 0 0 880 0 464m 3.35g 31m 22.3 0 0|0 0|0 1m 109k 2 17:17:16 0 0 473 0 0 474 0 464m 3.35g 36m 17.2 0 0|0 0|0 1m 59k 1 17:17:17 Source DB: insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 0 0 1 0 56g 115g 1.86g 0 0 0|0 0|0 62b 1k 1 17:16:29 0 0 0 0 0 1 0 56g 115g 1.86g 0 0 0|0 0|0 62b 1k 1 17:16:30 1 1 1 0 1 3 0 56g 115g 1.87g 51.1 0 0|0 0|1 475b 697k 2 17:16:31 0 0 0 0 0 1 0 56g 115g 349m 129 0 0|0 0|1 62b 1k 2 17:16:32 0 0 0 0 0 1 0 56g 115g 367m 101 0 0|0 0|1 62b 1k 2 17:16:33 0 0 0 0 0 1 0 56g 115g 384m 73.4 0 0|0 0|1 62b 1k 2 17:16:34 0 0 0 0 0 1 0 56g 115g 428m 112 0 0|0 0|1 62b 1k 2 17:16:35 0 0 0 0 0 1 0 56g 115g 436m 94.8 0 0|0 0|1 62b 1k 2 17:16:36 0 0 0 0 0 1 0 56g 115g 450m 108 0 0|0 0|1 62b 1k 2 17:16:37 0 0 0 0 0 1 0 56g 115g 461m 110 0 0|0 0|1 62b 1k 2 17:16:38 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 0 0 1 0 56g 115g 480m 95.5 0 0|0 0|1 62b 1k 2 17:16:39 0 0 0 0 0 1 0 56g 115g 494m 73.5 0 0|0 0|1 62b 1k 2 17:16:40 0 0 0 0 0 1 0 56g 115g 531m 120 0 0|0 0|1 62b 1k 2 17:16:41 0 0 0 0 0 1 0 56g 115g 541m 108 0 0|0 0|1 62b 1k 2 17:16:42 0 0 0 0 0 1 0 56g 115g 564m 74.6 0 0|0 0|1 62b 1k 2 17:16:43 0 0 0 0 0 1 0 56g 115g 579m 127 0 0|0 0|1 62b 1k 2 17:16:44 0 0 0 0 0 1 0 56g 115g 605m 100 0 0|0 0|1 62b 1k 2 17:16:45 0 0 0 0 0 1 0 56g 115g 631m 98.5 0 0|0 0|1 62b 1k 2 17:16:46 0 0 0 0 0 1 0 56g 115g 657m 89.1 0 0|0 0|1 62b 1k 2 17:16:47 0 0 0 0 0 1 0 56g 115g 662m 87.8 0 0|0 0|1 62b 1k 2 17:16:48 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 0 0 1 0 56g 115g 680m 93.5 0 0|0 0|1 62b 1k 2 17:16:49 0 1 0 530 1 531 0 56g 115g 767m 81.3 0 0|0 0|1 72k 4m 2 17:16:50 0 0 0 884 0 886 0 56g 115g 721m 48 0 0|0 0|0 120k 74k 2 17:16:51 0 0 0 1079 0 1080 0 56g 115g 698m 33.6 0 0|0 0|0 146k 90k 2 17:16:52 0 0 0 1294 0 1294 0 56g 115g 720m 23.8 0 0|1 0|1 175k 108k 2 17:16:53 0 0 0 1118 1 1120 0 56g 115g 740m 31.9 0 0|0 0|0 152k 4m 2 17:16:54 0 0 0 1111 0 1112 0 56g 115g 718m 33.3 0 0|0 0|0 151k 93k 2 17:16:55 0 0 0 1097 0 1098 0 56g 115g 706m 33.1 0 0|0 0|0 149k 92k 2 17:16:56 0 0 0 1097 0 1098 0 56g 115g 701m 33.2 0 0|0 0|0 149k 92k 2 17:16:57 0 0 0 1091 1 1092 0 56g 115g 696m 35.1 0 0|0 0|0 148k 4m 2 17:16:58 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 1149 0 1150 0 56g 115g 718m 21.3 0 0|0 0|0 156k 96k 2 17:16:59 0 0 0 1091 0 1091 0 56g 115g 707m 31.2 0 0|0 0|1 148k 91k 2 17:17:00 0 0 0 468 0 470 0 56g 115g 714m 9.3 0 0|0 0|0 63k 40k 2 17:17:01 0 0 0 332 0 333 0 56g 115g 694m 27.3 0 0|0 0|0 45k 28k 2 17:17:02 0 0 0 1013 1 1014 0 56g 115g 713m 17.8 0 0|0 0|0 137k 4m 2 17:17:03 0 0 0 551 0 552 0 56g 115g 691m 39.9 0 0|0 0|0 74k 47k 2 17:17:04 0 0 0 1140 0 1141 0 56g 115g 709m 19.7 0 0|0 0|0 155k 95k 2 17:17:05 0 0 0 126 0 127 0 56g 115g 709m 2.1 0 0|0 0|0 17k 11k 2 17:17:06 0 0 0 92 0 92 0 56g 115g 710m 1.6 0 0|0 0|1 12k 8k 2 17:17:07 0 0 0 605 1 607 0 56g 115g 700m 29.1 0 0|0 0|0 82k 4m 2 17:17:08 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 814 0 814 0 56g 115g 699m 17.3 0 0|0 0|1 110k 68k 2 17:17:09 0 0 0 1036 0 1037 0 56g 115g 708m 18.4 0 0|0 0|0 140k 87k 2 17:17:10 0 0 0 820 0 821 0 56g 115g 715m 14.4 0 0|1 0|1 111k 69k 2 17:17:11 0 0 0 298 0 300 0 56g 115g 683m 24.2 0 0|0 0|0 40k 26k 2 17:17:12 0 0 0 907 1 908 0 56g 115g 699m 20.4 0 0|0 0|0 123k 4m 2 17:17:13 0 0 0 946 0 947 0 56g 115g 707m 16.4 0 0|0 0|0 128k 79k 2 17:17:14 0 0 0 374 0 375 0 56g 115g 677m 36.7 0 0|0 0|0 50k 32k 2 17:17:15 0 0 0 905 1 905 0 56g 115g 689m 16.2 0 0|0 0|0 123k 821k 2 17:17:16 0 0 0 259 0 261 0 56g 115g 690m 5.1 0 0|0 0|0 35k 22k 1 17:17:17 0 0 0 0 0 1 0 56g 115g 690m 0 0 0|0 0|0 62b 1k 1 17:17:18 Total: MongoDB shell version: 2.0.1 connecting to: localhost:27017/prodcopy archiving clicks for ad_id: 4d6c12497b90420373000062 ensuring sparse index on field "arch" marking documents for archive... queued 22227 documents for archive in 19369ms. archiving queued documents to archive.clicks... archived 22227 queued documents to archive.clicks in 26901ms. Upgrading the hard drive to an SSD, I was able to almost triple the performance of the archiving step, and archive ~1300 documents per second. Obviously this is all a giant hack, and it would be nice if MongoDB supported some kind of partitioning feature so we didn’t have to do any of this manually. But until then, this is the best I could get given the tools currently available. Source: http://blog.brianploetz.com/post/12131083486/adventures-in-archiving-with-mongodb
November 21, 2011
by Mitch Pronschinske
· 14,502 Views
article thumbnail
Freight Management System on NetBeans
Lynden is a family of transportation and logistics companies specialized in shipping to Alaska and other locations worldwide. Over land, on the water, in the air - or in any combination - Lynden has been helping customers solve transportation problems for over a century. The Lynden Freight Management System is a NetBeans Platform application which serves a dual purpose as both a planning and freight tracking tool. The Planning module allows terminal managers to see all freight that is currently inbound to their location as well as freight that is scheduled to depart from their location so they can make the most efficient use of their dock space and resources as possible. The Trace module allows customer service personnel to search for customer account information, view the tracking history of any given freight item in the system as well as display any documents related to the shipment, such as bills of lading or delivery receipts. NetBeans Platform Lynden has benefited from the NetBeans Platform as it allows developers to focus on the business logic of our applications rather than the underlying "plumbing". We are able to leverage built-in support for event handling, enable/disable functionality on UI controls, dockable windows, and automatic updates for our application with minimal work compared to rolling our own framework. We chose to go the desktop application route as we have a number of existing desktop applications here that this application will likely need to interface with at some point, as well as a commercial set of rich UI components that we have been using for some time now. For the initial deployment, we will be pushing the installer out to employee PCs via the Landesk remote desktop administration tool. Future updates to various modules within the application will be done via the update center functionality built into the NetBeans Platform. Screenshots
November 19, 2011
by Rob Terpilowski
· 11,877 Views · 3 Likes
article thumbnail
Tackling the Circular Dependency in Java...
Let me first define what we mean by circular dependency in OOAD terms vis-a-vis Java. Suppose we have a class called A which has class B’s Object. (in UML terms A HAS B). at the same time we class B is also composed of Object of class A (in UML terms B HAS A). obviously this represents circular dependency because while creating the object of A, the compiler must know the size of B... on the other hand while creating object of B, the compiler must know the size of A. this is something like egg vs. chicken problem... this may be possible in real life situation as well. for example suppose a multi storied building has a lift. so in the UML terms, the building HAS lift... but at the same time, suppose, while constructing the lift object, we need to give it the information about the building object to access various functionalities of the Building class... for example, suppose the speed of the lift is set depending on the number of floors of the Building... hence while constructing the Lift object it must access the functionalities of the Building object which will give the number of floors the building has got...hence in UML terms the lift HAS building... so this is a sure case of circular dependency... in real java code it will look something as follows: public class Building { private Lift lift; private int floor; public Building(){ lift = new Lift(); setFloor(15); } public int getFloor(){ return floor; } public void setFloor(int floor){ this.floor = floor; } }//end of class building //class Lift public class Lift { private Building building; private int Speed; public Lift(){ building = new Building(); setSpeed(); } public void setSpeed(){ if (building.getFloor()>20){ //one set of functionalities //may be the the speed of the lift will be more this.Speed = 10; } else { //different set of functionalities //may be the speed of the lift will be less this.Speed = 5; } } public int getSpeed(){ return Speed; } }//end of class Lift As it becomes clear from the above code, that while creating the Building object it will create the Lift object, and while creating the Lift object, it will try to create a Building object to access some of its functionalities... So, ultimately it will go out of memory and we get a StackOverflow runtime exception... So how do we handle this problem in Java? We actually tackle this problem by declaring an IBuildingProxy interface and by deriving our Building class from that... the lift class, instead of Having Building object, it Has IBuildingProxy... the source code of the solution looks like the following... public interface IBuildingProxy { int getFloor(); void setFloor(int floor); } public class Building implements IBuildingProxy{ private Lift lift; private int floor; public Building(){ lift = new Lift(this); setFloor(15); } public int getFloor(){ return floor; } public void setFloor(int floor){ this.floor = floor; } } public class Lift { private IBuildingProxy building; private int Speed; public Lift(Building b){ this.building = b; setSpeed(); } private void setSpeed(){ if (building.getFloor()>20){ / /one set of functionalities //may be the the speed of the lift will be more this.Speed = 10; } else { //different set of functionalities //may be the speed of the lift will be less this.Speed = 5; } } public int getSpeed(){ return Speed; } } public class CircularDependencyTest { public static void main(String[] args){ Building b = new Building(); Lift l = new Lift(b); } } So whats the principle behind such work around... It will be clear soon... As it becomes clear from the code that Building HAS Lift... That is not a problem... Now when it comes to solve the part that Lift HAS Building, instead of the Building object, we have created an IBuildingProxy interface and we pass it to the Lift class... what it essentially means, that the building class knows the memory requirement to initialize the Lift object, and as the Lift class HAS just a proxy interface of the Building, it does not have to care for the Building's memory requirement... and that solves the problem... Hope this discussion becomes helpful for Java learners...
November 17, 2011
by Somenath Mukhopadhyay
· 32,603 Views · 2 Likes
article thumbnail
ASP.NET MVC: Converting business objects to select list items
Some of our business classes are used to fill dropdown boxes or select lists. And often you have some base class for all your business classes. In this posting I will show you how to use base business class to write extension method that converts collection of business objects to ASP.NET MVC select list items without writing a lot of code. BusinessBase, BaseEntity and other base classes I prefer to have some base class for all my business classes so I can easily use them regardless of their type in contexts I need. NB! Some guys say that it is good idea to have base class for all your business classes and they also suggest you to have mappings done same way in database. Other guys say that it is good to have base class but you don’t have to have one master table in database that contains identities of all your business objects. It is up to you how and what you prefer to do but whatever you do – think and analyze first, please. :) To keep things maximally simple I will use very primitive base class in this example. This class has only Id property and that’s it. public class BaseEntity { public virtual long Id { get; set; } } Now we have Id in base class and we have one more question to solve – how to better visualize our business objects? To users ID is not enough, they want something more informative. We can define some abstract property that all classes must implement. But there is also another option we can use – overriding ToString() method in our business classes. public class Product : BaseEntity { public virtual string SKU { get; set; } public virtual string Name { get; set; } public override string ToString() { if (string.IsNullOrEmpty(Name)) return base.ToString(); return Name; } } Although you can add more functionality and properties to your base class we are at point where we have what we needed: identity and human readable presentation of business objects. Writing list items converter Now we can write method that creates list items for us. public static class BaseEntityExtensions { public static IEnumerable ToSelectListItems (this IList baseEntities) where T : BaseEntity { return ToSelectListItems((IEnumerator) baseEntities.GetEnumerator()); } public static IEnumerable ToSelectListItems (this IEnumerator baseEntities) { var items = new HashSet(); while (baseEntities.MoveNext()) { var item = new SelectListItem(); var entity = baseEntities.Current; item.Value = entity.Id.ToString(); item.Text = entity.ToString(); items.Add(item); } return items; } } You can see here to overloads of same method. One works with List and the other with IEnumerator. Although mostly my repositories return IList when querying data there are always situations where I can use more abstract types and interfaces. Using extension methods in code In your code you can use ToSelectListItems() extension methods like shown on following code fragment. ... var model = new MyFormModel(); model.Statuses = _myRepository.ListStatuses().ToSelectListItems(); ... You can call this method on all your business classes that extend your base entity. Wanna have some fun with this code? Write overload for extension method that accepts selected item ID.
November 11, 2011
by Gunnar Peipman
· 13,760 Views
article thumbnail
Creating a backup in Team Foundation Server 2010 using the Power Tools
over the last few years the product team has been putting their finishing touches on a backup module for the team foundation server administration console. why you might ask do you need another way to backup? surely you can just backup the bits? well, you could, but as tfs has a lot of moving parts it can get very complicated to creating a backup . required permissions identify databases create tables in databases create a stored procedure for marking tables create a stored procedure for marking all tables at once create a stored procedure to automatically mark tables create a scheduled job to run the table-marking procedure create a maintenance plan for full backups create a maintenance plan for differential backups create a maintenance plan for transaction backups back up additional lab management components -from “back up team foundation server” on msdn there are a heck of a lot of databases that, depending on your environment, might be spread over your entire network. figure: deployment topologies (where is my data?) from msdn so, how is this problem solved. well the tfs team have create a tool to create all of the backups and all of the job as well as managing the backup location for you. this sounds fantastic, but how about in practice. was it really that easy? well….not really…here is the extra stuff i found out: your account must own the share owning the folder does not cut it (see error #1- tf254027). sql must be running under a domain account or network service sql must also have permission to the share, and the validation will get confused if you use “localsystem” instead of network service or a domain account (see error #2- tf254027) the account running sql must have permission to create spn’s the account that is used for sql must be able to both see and create service principal names in active directory (see error #3: terminating your tfs server). once you learn how to google without keywords and read your servers mind you will have a nice backup system going… error #1- tf254027 i initially got an error because the accounts did not really have full control over the target location. this is a problem with the share. although i have full permission for \\fileserver1\share\tfsbackups it is just a folder under the \\fileserver1\share\ location and i do not have permission to change the sharing settings there. figure: tf254027 is caused by permission issues [info @16:36:34.342] granting account root_company\tfssqlbox$ permission on folder \\fileserver1\share\tfsbackups [info @16:36:34.348] system.unauthorizedaccessexception: attempted to perform an unauthorized operation. at system.security.accesscontrol.win32.setsecurityinfo(resourcetype type, string name, safehandle handle, securityinfos securityinformation, securityidentifier owner, securityidentifier group, genericacl sacl, genericacl dacl) at system.security.accesscontrol.nativeobjectsecurity.persist(string name, safehandle handle, accesscontrolsections includesections, object exceptioncontext) at system.security.accesscontrol.filesystemsecurity.persist(string fullpath) at microsoft.teamfoundation.powertools.admin.helpers.filehelper.grantfolderpermission(string account, string path) [info @16:36:34.350] granting account root_company\tfs.services permission on folder \\fileserver1\share\tfsbackups [info @16:36:34.352] system.unauthorizedaccessexception: attempted to perform an unauthorized operation. at system.security.accesscontrol.win32.setsecurityinfo(resourcetype type, string name, safehandle handle, securityinfos securityinformation, securityidentifier owner, securityidentifier group, genericacl sacl, genericacl dacl) at system.security.accesscontrol.nativeobjectsecurity.persist(string name, safehandle handle, accesscontrolsections includesections, object exceptioncontext) at system.security.accesscontrol.filesystemsecurity.persist(string fullpath) at microsoft.teamfoundation.powertools.admin.helpers.filehelper.grantfolderpermission(string account, string path) [error @16:36:34.352] granting permission to account root_company\tfssqlbox$ on path \\fileserver1\share\tfsbackups failed figure: the log files get to the root of the problem, but not the reason after much messing around i have found that you can’t use a sub-folder of a share that you do not have permission for. you require permission to the share itself to apply permissions. error #2- tf254027 lets try this again with a share that we control. i will create a backup share on the tfs server and at least then i control then permissions. figure: the next error looks the same, but it is subtly different [info @18:12:05.813] "verify: grant backup plan permissions\root\verifybackuppathpermissionsgrantedsuccessfully(verifybackuppathpermissionsgrantedsuccessfully): exiting verification with state completed and result success" [info @18:12:05.813] verify: grant backup plan permissions\root\verifydummybackupcreation(verifytestbackupcreatedsuccessfully): starting verification [info @18:12:05.813] verify test backup created successfully [info @18:12:05.813] starting creating backup test validation [error @18:12:06.132] microsoft.sqlserver.management.smo.failedoperationexception: backup failed for server 'sqlserver1'. ---> microsoft.sqlserver.management.common.executionfailureexception: an exception occurred while executing a transact-sql statement or batch. ---> system.data.sqlclient.sqlexception: cannot open backup device '\\tfsserver1\tfsbackup\temp_20111104111205.bak'. operating system error 5(access is denied.). backup database is terminating abnormally. at microsoft.sqlserver.management.common.connectionmanager.executetsql(executetsqlaction action, object execobject, dataset filldataset, boolean catchexception) at microsoft.sqlserver.management.common.serverconnection.executenonquery(string sqlcommand, executiontypes executiontype) --- end of inner exception stack trace --- at microsoft.sqlserver.management.common.serverconnection.executenonquery(string sqlcommand, executiontypes executiontype) at microsoft.sqlserver.management.common.serverconnection.executenonquery(stringcollection sqlcommands, executiontypes executiontype) at microsoft.sqlserver.management.smo.executionmanager.executenonquery(stringcollection queries) at microsoft.sqlserver.management.smo.backuprestorebase.executesql(server server, stringcollection queries) at microsoft.sqlserver.management.smo.backup.sqlbackup(server srv) --- end of inner exception stack trace --- at microsoft.sqlserver.management.smo.backup.sqlbackup(server srv) at microsoft.teamfoundation.powertools.admin.helpers.backupfactory.testbackupcreation(string path) [error @18:12:06.184] !verify error!: account root_comapny\martin.hinshelwood failed to create backups using path \\tfsserver1\tfsbackup [info @18:12:06.184] "verify: grant backup plan permissions\root\verifydummybackupcreation(verifytestbackupcreatedsuccessfully): exiting verification with state completed and result error" [info @18:12:06.184] !verify result!: 5 completed, 0 skipped: 4 success, 1 errors, 0 warnings [info @18:12:06.197] verify: backup tasks verifications(vcontainer): starting verification [info @18:12:06.197] a generic container node that does not contribute to results [info @18:12:06.197] "verify: backup tasks verifications(vcontainer): exiting verification with state ignore and result ignore" [info @18:12:06.197] verify: backup tasks verifications\root(vcontainer): starting verification [info @18:12:06.197] a generic container node that does not contribute to results [info @18:12:06.197] "verify: backup tasks verifications\root(vcontainer): exiting verification with state ignore and result ignore" [info @18:12:06.197] verify: backup tasks verifications\root\verifydummybackupcreation(verifytestbackupcreatedsuccessfully): starting verification [info @18:12:06.197] verify test backup created successfully [info @18:12:06.197] starting creating backup test validation [error @18:12:06.389] microsoft.sqlserver.management.smo.failedoperationexception: backup failed for server sqlserver1'. ---> microsoft.sqlserver.management.common.executionfailureexception: an exception occurred while executing a transact-sql statement or batch. ---> system.data.sqlclient.sqlexception: cannot open backup device '\\tfsserver1\tfsbackup\temp_20111104111206.bak'. operating system error 5(access is denied.). backup database is terminating abnormally. at microsoft.sqlserver.management.common.connectionmanager.executetsql(executetsqlaction action, object execobject, dataset filldataset, boolean catchexception) at microsoft.sqlserver.management.common.serverconnection.executenonquery(string sqlcommand, executiontypes executiontype) --- end of inner exception stack trace --- at microsoft.sqlserver.management.common.serverconnection.executenonquery(string sqlcommand, executiontypes executiontype) at microsoft.sqlserver.management.common.serverconnection.executenonquery(stringcollection sqlcommands, executiontypes executiontype) at microsoft.sqlserver.management.smo.executionmanager.executenonquery(stringcollection queries) at microsoft.sqlserver.management.smo.backuprestorebase.executesql(server server, stringcollection queries) at microsoft.sqlserver.management.smo.backup.sqlbackup(server srv) --- end of inner exception stack trace --- at microsoft.sqlserver.management.smo.backup.sqlbackup(server srv) at microsoft.teamfoundation.powertools.admin.helpers.backupfactory.testbackupcreation(string path) figure: this time the error is lying and is from sql not locally as it implies it looks like the problem is that sql server can’t write to that folder, but i can and the machine account can. lets try this from the sql server itself, and with a native backup. figure: sql server can’t write to that location dam… so even a native sql backup can’t write to this location. title: microsoft sql server management studio ------------------------------ backup failed for server 'sqlserver1'. (microsoft.sqlserver.smoextended) for help, click: http://go.microsoft.com/fwlink?prodname=microsoft+sql+server&prodver=10.50.2500.0+((kj_pcu_main).110617-0038+)&evtsrc=microsoft.sqlserver.management.smo.exceptiontemplates.failedoperationexceptiontext&evtid=backup+server&linkid=20476 ------------------------------ additional information: system.data.sqlclient.sqlerror: cannot open backup device '\\tfsserver1\tfsbackup\moo.bak'. operating system error 5(access is denied.). (microsoft.sqlserver.smo) for help, click: http://go.microsoft.com/fwlink?prodname=microsoft+sql+server&prodver=10.50.2500.0+((kj_pcu_main).110617-0038+)&linkid=20476 ------------------------------ buttons: ok ------------------------------ figure: sql server errors suck even more as it turns out, sql server is running under “localserivce” which is not authenticating against our share. so we need to change the service that tfs runs under. error #3: terminating your tfs server as we should always use the sql server configuration manager to change these things i fired it up and since i already have a domain account for running tfs under i decided to use that one. figure: this is easy when you apply it will ask you to restart sql, but it should be all complete. lets check tfs and make sure that everything is running… figure: omg! what just happened! oh shit: i think i just broke tfs. why can’t tfs connect? lets try the sql management studio and see. figure: what is a sspi? this does not look good… after i have hastily changed the service account back to the original value and made use that his fixed tfs i wanted to also figure out why it broke. usually i would just ask shad (one of my extremely technical colleagues) but alas he is on his honeymoon. some googling turned up an spn issue. the account that sql runs under must be able to both read and write service principal names for itself in active directory. this can be set, but only be a domain admin. dynamically set spn’s for sql service accounts so lets go with network service instead. if we change the account that sql server runs under to “network service” then i can add permission for “root_company\sqlserver1$” to my share and get it working. yes, servers have ad accounts as well.
November 5, 2011
by Martin Hinshelwood
· 10,114 Views
article thumbnail
Recommendation Engine Models
In a classical model of recommendation system, there are "users" and "items". User has associated metadata (or content) such as age, gender, race and other demographic information. Items also has its metadata such as text description, price, weight ... etc. On top of that, there are interaction (or transaction) between user and items, such as userA download/purchase movieB, userX give a rating 5 to productY ... etc. Now given all the metadata of user and item, as well as their interaction over time, can we answer the following questions ... What is the probability that userX purchase itemY ? What rating will userX give to itemY ? What is the top k unseen items that should be recommended to userX ? Content-based Approach In this approach, we make use of the metadata to categorize user and item and then match them at the category level. One example is to recommend jobs to candidates, we can do a IR/text search to match the user's resume with the job descriptions. Another example is to recommend an item that is "similar" to the one that the user has purchased. Similarity is measured according to the item's metadata and various distance function can be used. The goal is to find k nearest neighbors of the item we know the user likes. Collaborative Filtering Approach In this approach, we look purely at the interactions between user and item, and use that to perform our recommendation. The interaction data can be represented as a matrix. Notice that each cell represents the interaction between user and item. For example, the cell can contain the rating that user gives to the item (in the case the cell is a numeric value), or the cell can be just a binary value indicating whether the interaction between user and item has happened. (e.g. a "1" if userX has purchased itemY, and "0" otherwise. The matrix is also extremely sparse, meaning that most of the cells are unfilled. We need to be careful about how we treat these unfilled cells, there are 2 common ways ... Treat these unknown cells as "0". Make them equivalent to user giving a rate "0". This may or may not be a good idea depends on your application scenarios. Guess what the missing value should be. For example, to guess what userX will rate itemA given we know his has rate on itemB, we can look at all users (or those who is in the same age group of userX) who has rate both itemA and itemB, then compute an average rating from them. Use the average rating of itemA and itemB to interpolate userX's rating on itemA given his rating on itemB. User-based Collaboration Filter In this model, we do the following Find a group of users that is “similar” to user X Find all movies liked by this group that hasn’t been seen by user X Rank these movies and recommend to user X This introduces the concept of user-to-user similarity, which is basically the similarity between 2 row vectors of the user/item matrix. To compute the K nearest neighbor of a particular users. A naive implementation is to compute the "similarity" for all other users and pick the top K. Different similarity functions can be used. Jaccard distance function is defined as the number of intersections of movies that both users has seen divided by the number of union of movies they both seen. Pearson similarity is first normalizing the user's rating and then compute the cosine distance. There are two problems with this approach Compare userX and userY is expensive as they have millions of attributes Find top k similar users to userX require computing all pairs of userX and userY Location Sensitive Hashing and Minhash To resolve problem 1, we approximate the similarity using a cheap estimation function, called minhash. The idea is to find a hash function h() such that the probability of h(userX) = h(userY) is proportion to the similarity of userX and userY. And if we can find 100 of h() function, we can just count the number of such function where h(userX) = h(userY) to determine how similar userX is to userY. The idea is depicted as follows ... It will be expensive to permute the rows if the number of rows is large. Remember that the purpose of h(c1) is to return row number of the first row that is 1. So we can scan each row of c1 to see if it is 1, if so we apply a function newRowNum = hash(rowNum) to simulate a permutation. Take the minimum of the newRowNum seen so far. As an optimization, instead of doing one column at a time, we can do it a row at the time, the algorithm is as follows To solve problem 2, we need to avoid computing all other users' similarity to userX. The idea is to hash users into buckets such that similar users will be fall into the same bucket. Therefore, instead of computing all users, we only compute the similarity of those users who is in the same bucket of userX. The idea is to horizontally partition the column into b bands, each with r rows. By pick the parameter b and r, we can control the likelihood (function of similarity) that they will fall into the same bucket in at least one band. Item-based Collaboration Filter If we transpose the user/item matrix and do the same thing, we can compute the item to item similarity. In this model, we do the following ... Find the set of movies that user X likes (from interaction data) Find a group of movies that is similar to these set of movies that we know user X likes Rank these movies and recommend to user X It turns out that computing item-based collaboration filter has more benefit than computing user to user similarity for the following reasons ... Number of items typically smaller than number of users While user's taste will change over time and hence the similarity matrix need to be updated more frequent, item to item similarity tends to be more stable and requires less update. Singular Value Decomposition If we look back at the matrix, we can see the matrix multiplication is equivalent to mapping an item from the item space to the user space. In other words, if we view each of the existing item as an axis in the user space (notice, each user is a vector of their rating on existing items), then multiplying a new item with the matrix gives the same vector like the user. So we can then compute a dot product with this projected new item with user to determine its similarity. It turns out that this is equivalent to map the user to the item space and compute a dot product there. In other words, multiply the matrix is equivalent to mapping between item space and user space. Now lets imagine there is a hidden concept space in between. Instead of jumping directly from user space to item space, we can think of jumping from user space to a concept space, and then to the item space. Notice that here we first map the user space to the concept space and also map the item space to the concept space. Then we match both user and item at the concept space. This is a generalization of our recommender. We can use SVD to factor the matrix into 2 parts. Let P be the m by n matrix (m rows and n columns). P = UDV where U is an m by m matrix, each column represents the eigenvectors of P*transpose(P). And V is an n by n matrix with each row represents the eigenvector of transpose(P)*P. D is a diagonal matrix containing eigenvalues of P*transpose(P), or transpose(P)*P. In other words, we can decompose P into U*squareroot(D) and squareroot(D)*V. Notice that D can be thought as the strength of each "concept" in the concept space. And the value is order in terms of their magnitude in decreasing order. If we remove some of the weakest concept by making them zero, we reduce the number of non-zero elements in D, which effective generalize the concept space (make them focus in the important concepts). Calculate SVD decomposition for matrix with large dimensions is expensive. Fortunately, if our goal is to compute an SVD approximation (with k diagonal non-zero value), we can use the random projection mechanism as describer here. Association Rule Based In this model, we use the market/basket association rule algorithm to discover rule like ... {item1, item2} => {item3, item4, item5} We represent each user as a basket and each viewing as an item (notice that we ignore the rating and use a binary value). After that we use association rule mining algorithm to detect frequent item set and the association rules. Then for each user, we match the user's previous viewing items to the set of rules to determine what other movies should we recommend. Evaluate the recommender After we have a recommender, how do we evaluate the performance of it ? The basic idea is to use separate the data into the training set and the test set. For the test set, we remove certain user-to-movies interaction (change certain cells from 1 to 0) and pretending the user hasn't seen the item. Then we use the training set to train a recommender and then fit the test set (with removed interaction) to the recommender. The performance is measured by how much overlap between the recommended items with the one that we have removed. In other words, a good recommender should be able to recover the set of items that we have removed from the test set. Leverage tagging information on items In some cases, items has explicit tags associated with them (we can considered the tags is a user-annotated concept space added to the items). Consider each item is described with a vector of tags. Now user can also be auto-tagged based on the items they have interacted. For example, if userX purchase itemY which is tagged with Z1, and Z2. Then user will increase her tag Z1 and Z2 in her existing tag vector. We can use a time decay mechanism to update the user's tag vector as follows ... current_user_tag = alpha * item_tag + (1 - alpha) * prev_user_tag To recommend an item to the user, we simply need to calculate the top k items by computing the dot product (ie: cosine distance) of the user tag vector and the item tag vector. Source: http://horicky.blogspot.com/2011/09/recommendation-engine.html
November 2, 2011
by Ricky Ho
· 27,070 Views · 2 Likes
article thumbnail
Avoid Lazy JPA Collections
Hibernate (and actually JPA) has collection mappings: @OneToMany, @ManyToMany, @ElementCollection. All of these are by default lazy. This means the collections are specific implementations of the List or Set interface that hold a reference to the persistent session and the values are loaded from the database only if the collection is accessed. That saves unnecessary database queries if you only occasionally use the collection. However, there’s a problem with that. The problem that manifests itself through the exception that in my observations is 2nd most commonly asked exception (after NullPointerException) – the LazyInitializationException. The problem is that the session is usually open for your service layer and is closed as soon as you return the entity to the view layer. And when you try to iterate the uninitialized collection in your view (jsp for example), the collection throws LazyInitializationException, because the session that they hold a reference to is already closed and they can’t fetch the items. How is this solved? The so called OpenSessionInView / OpenEntityManagerInView “patterns”. In short: you make a filter that opens the session when the request starts and closes it after the view has been rendered (and not after the service layer finishes). Some people call that an anti-pattern, because it leaks persistence handling into the view layer, and complicates the setup. I wouldn’t say it’s that bad: generally it solves the problem without introducing other problems. But in all recent project I’ve been involved, we aren’t using OpenSessionInView, and it works fine. It works fine because we aren’t using lazy collections. But then, you’ll rightly point, you will be fetching “the whole world” when you load a single entity. Well, no. There are two types of *ToMany mappings: value-type mappings where the collection logically does not hold more than a dozen elements. This is in most cases @ElementCollection, and also @*ToMany with items like “Category” or “Price” that are just more complex value objects, but that do not hold any other mappings themselves. Another common feature of these types of collections is that they are usually displayed in the UI together with their owning entity. It is most likely that you want to display the categories of an article, for example. For this type of collections EAGER is the better option. You’ll have to fetch them anyway, why not let hibernate (or any jpa implementation) think of some clever join? As I said – the collections are logically not bigger than a dozen or two, so fetching them won’t be a performance hit. And, logically, they won’t fetch a big object graph with them. mappings across the big, core entities. This can be “all orders made by the user” or “all users for the organization”, “all items of the supplier”, etc. You certainly don’t want to fetch them eagerly. Because if you fetch 2000 users for an organization, which in turn have 1000 orders each, and an order has 3 items on average which in turn have a collection of all people who have purchased it.. you’ll end up with your entire database in memory. Obviously you need lazy collections, right? Well, no. In that case you should not be using collection mappings at all. These types of relations are, in 99% of the cases, displayed in paged lists in the UI. Or in search results. They are never (and should never) be displayed all on one screen (or should rarely be returned in one API call, if your application provides something like a REST API). You have to make queries for them, and use query.setMaxResults and query.setFirstResult() (or limit them with some restrictive criteria). Furthermore having the collections mapped means someone will try to use them at some point, which may fail. And if the object is serialized (xml, json, etc.) the collection contents will be fetched. Something you almost certainly don’t want to happen. (A draft idea here: JPA could have a PagedList collection that would allow paged lazy fetching, thus eliminating the need for a query) So what did I just say – that you should never use lazy collections. Use eager collections for very simple, shallow mappings, and use paged queries for the bigger ones. Well, not exactly. Lazy collections are there and they have application, though it is rather limited. Or at least they are way less applicable than they are used. Here’s an example scenario where I found it applicable. In my side-project I have a Message entity, and it holds a collection of Picture entities. When a user uploads a picture, it is stored in that collection. A message can have no more than 10 pictures, so the collection could very well be eager. But then, Message is the most commonly used entity – it’s fetched virtually on every request. But only some messages have pictures (how many of the tweets on your stream have a a picture upload?). So I don’t want hibernate to make queries just to find out there are no pictures for a given message. Hence I store the number of pictures in a separate field, make the pictures collection lazy, and Hibernate.initialize(..) it manually only if the number of pictures is > 0. So there are scenarios, when the entity has optional collections that fall into the first category above (“small, shallow collections”). So if it is small, shallow and optional (say, used in less than 20% of the cases), then you should go with Lazy to save unnecessary queries. For everything else – having lazy collections will make your life harder. From http://techblog.bozho.net/?p=645
October 28, 2011
by Bozhidar Bozhanov
· 21,637 Views · 1 Like
  • Previous
  • ...
  • 887
  • 888
  • 889
  • 890
  • 891
  • 892
  • 893
  • 894
  • 895
  • 896
  • ...
  • Next
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook
×