Frequently developers rely on 3d party libraries to avoid reinventing the wheel, particularly in the Java world, with projects like Apache and Spring so prevalent. When dealing with these frameworks, we often have little or no control of the behaviour of their classes. This can sometimes lead to problems. For instance, if you want to deep clone an object that doesn’t provide a suitable clone method, what are your options, short of writing a bunch of code? Clone through Serialization The simplest approach is to clone by taking advantage of an object being Serializable. Apache Commons provides a method to do this, but for completeness, code to do it yourself is below also. @SuppressWarnings("unchecked") public static T cloneThroughSerialize(T t) throws Exception { ByteArrayOutputStream bos = new ByteArrayOutputStream(); serializeToOutputStream(t, bos); byte[] bytes = bos.toByteArray(); ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(bytes)); return (T)ois.readObject(); } private static void serializeToOutputStream(Serializable ser, OutputStream os) throws IOException { ObjectOutputStream oos = null; try { oos = new ObjectOutputStream(os); oos.writeObject(ser); oos.flush(); } finally { oos.close(); } } // using our custom method Object cloned = cloneThroughSerialize (someObject); // or with Apache Commons cloned = org.apache.commons.lang. SerializationUtils.clone(someObject); But what if the class we want to clone isn’t Serializable and we have no control over the source code or can’t make it Serializable? Option 1 – Java Deep Cloning Library There’s a nice little library which can deep clone virtually any Java Object – cloning. It takes advantage of Java’s excellent reflection capabilities to provide optimized deep-cloned versions of objects. Cloner cloner=new Cloner(); Object cloned = cloner.deepClone(someObject); As you can see, it’s very simple and effective, and requires minimal code. It has some more advanced abilities beyond this simple example, which you can check out here. Option 2 – JSON Cloning What about if we are not able to introduce a new library to our codebase? Some of us deal with approval processes to introduce new libraries, and it may not be worth it for a simple use case. Well, as long as we have some way to serialize and restore an object, we can make a deep copy. JSON is commonly used, so it’s a good candidate,since most of us use one JSON library or another. Most JSON libraries in Java have the ability to effectively serialize any POJO without any configuration or mapping required. This means that if you have a JSON library and cannot or will not introduce more libraries to provide deep cloning, you can leverage an existing JSON library to get the same effect. Note this method will be slower than others, but for the vast majority of applications, this won’t cause any performance problems. Below is an example using the GSON library. @SuppressWarnings("unchecked") public static T cloneThroughJson(T t) { Gson gson = new Gson(); String json = gson.toJson(t); return (T) gson.fromJson(json, t.getClass()); } // ... Object cloned = cloneThroughJson(someObject); Note that this is likely only to work if the copied object has a default no-argument constructor. In the case of GSON, you can use an instance creator to get around this. Other frameworks have similar concepts, so you can use that if you hit an issue with an unmodifiable class having not having the default constructor. Conclusion One thing I do recommend is that for any classes you need to clone, you should add some unit tests to ensure everything behaves as expected. This can prevent changes to the classes (e.g. upgrading library versions) from breaking your application without your knowledge, especially if you have a continuous integration environment set up. I’ve outlined a couple methods to clone an object outside of normal cases without any custom code. If you’ve used any other methods to get the same result, please share. From http://www.carfey.com/blog/easy-deep-cloning-of-serializable-and-non-serializable-objects-in-java/

That’s an old trick, but it helped me today, and I’m pretty sure I’ll use it again. You can setup some System properties with Spring. Mix that with Maven resources filtering and you can execute your application with some specific System properties according to your profile. In your applicationContext.xml, add: logsDir ${logs.dir} Here ${logs.dir} is replaced before the execution of the application. During the build phase, maven will look at the properties you’ll have defined in your profile, and use them to filter the files you will have set (in this example, we set maven to filter all files in src/main/resources, where applicationContext.xml is located. For instance, we can define a property: logs.dir=/mydevmachine/logs for a dev profile, and logs.dir=/myprodmachine/logs for a prod profile. Then in your application, doing a System.getProperty(“logsDir”) will give you the correct value according to your maven profile. No need to add System properties to MAVEN_OPTS anymore! From http://jsoftbiz.wordpress.com/2011/12/08/use-spring-to-create-system-properties/

In this post I am going to explain URL rewriting in greater details. This post will contain basic of URL rewriting and will explain how we can do URL rewriting in fewer lines of code. Why we need URL rewriting? Let’s consider a simpler scenario we want to display a customer details on a ASP.NET Page so how our page will know that for which customer we need to display details? The simplest way of doing is to use query string we will pass a customer id which uniquely identifies customer in query string. So our url will look like this. Customer.aspx?Id=1 This will work but the problem with above URL is that its not user friendly and search engine friendly. who is going to remember that what query string parameter I am going to pass and why we need that parameter. Also when search engine will crawl this site it will going to read this URL blindly as this url is not informative because it query string is not readable for search engine crawlers. So your search engine will be ranked lower as this URL is not readable to search engine crawlers.Now when do a URL rewriting our URL will be cleaner shorter and simpler like this. Customers/Id/1/ Here anybody in world can understand it talking about customer and this page will used to show customer details.Even search engine crawler will also know that you are talking about customers. That is why we need URL rewriting. URL rewriting and ASP.NET In earlier versions of ASP.NET we have to write lots of code for URL rewriting but Now with ASP.NET 4.0 you can easily rewrite in fewer lines of code. So let’s start a Demo where I will demonstrate you how we can easily rewrite URLs. So let’s first create a ASP. NET web form application via File->New->Project and a dialog box will open just like below and then created a empty project called URL rewriting. After creating a project I have added global.asax – where we are going to write url mapping logic and then I have added an asp.net page called which will display customer information just like below. So everything is now ready let’s start writing code. First thing we need to do is to define routes. Route will map a URL to physical page.First I will create static function called Register route which map route to particular file and then I am going to call this from application_start event of global.asax. Following is code for that. using System; using System.Web.Routing; namespace UrlRewriting { public class Global : System.Web.HttpApplication { protected void Application_Start(object sender, EventArgs e) { RegisterRoutes(RouteTable.Routes); } public static void RegisterRoutes(RouteCollection routeCollection) { routeCollection.MapPageRoute("RouteForCustomer", "Customer/{Id}", "~/Customer.aspx"); } } } Now as mapping code has been done let’s right code for customer.aspx page. I have have following code in page_load event of customer.aspx using System; namespace UrlRewriting { public partial class Customer : System.Web.UI.Page { protected void Page_Load(object sender, EventArgs e) { string id = Page.RouteData.Values["Id"].ToString(); Response.Write("Customer Details page"); Response.Write(string.Format("Displaying information for customer : {0}",id)); } } } Here in above code you can see that I am getting value from page route data and then just printing it. In real world it will fetch customer data from database and show customer details on page. Now let’s run that application. It will print a details of customer as I have passed Id in URL suppose you pass 1 as id in URL then it will look like following. Now if you put 2 in url it will print information about customer 2. That’s it. So we have enabled URL rewriting in asp.net in fewer lines of code. In next post I am going to explain redirection with URL rewriting. Hope you like this post. Stay tuned for more.. Till then Happy programming

JSR 303 (Bean Validation) is the specification of the Java API for JavaBean validation in Java EE and Java SE. Simply put it provides an easy way of ensuring that the properties of your JavaBean(s) have the right values in them. This post aims to show you how to use the Bean Validation API in your project. To begin, imagine that you were building the next Facebook and you would need member(s) to register to use your application. In order to successfully register, prospective member(s) have to provide the following: a last name, a first name, a gender, an email address and a date of birth. In addition, the individual who is registering must be between 18 and 150 years inclusive. Prior to JSR 303, you probably would have needed a bunch of if-else statements to achieve the above requirements. Thankfully, not any more. We will begin by creating a JavaBean named 'Member' that would hold all the properties we are interested in. package validationapiblog.model; import java.util.Date; import validationapiblog.enums.Gender; /** * * @author Adedayo Ominiyi */ public class Member { private String lastName = null; private String firstName = null; private Gender gender = null; private String emailAddress = null; private Date dateOfBirth = null; public Member() { } public String getFirstName() { return firstName; } public void setFirstName(String firstName) { this.firstName = firstName; } public Gender getGender() { return gender; } public void setGender(Gender gender) { this.gender = gender; } public String getLastName() { return lastName; } public void setLastName(String lastName) { this.lastName = lastName; } public Date getDateOfBirth() { return dateOfBirth; } public void setDateOfBirth(Date dateOfBirth) { this.dateOfBirth = dateOfBirth; } public Integer getAge() { if (this.dateOfBirth != null) { // calculate age of member here } return null; } public String getEmailAddress() { return emailAddress; } public void setEmailAddress(String emailAddress) { this.emailAddress = emailAddress; } } The gender property is a simple enum and is shown below package validationapiblog.enums; /** * * @author Adedayo Ominiyi */ public enum Gender { MALE, FEMALE; } Now that we have the pieces to the puzzle. The next step is to download an implementation of JSR 303. For this post we would be using the reference implementation namely Hibernate Validator. The version as at the time this post was written is 4.2.0 Final. After downloading it you should add the following 4 jars to the classpath of your project: hibernate-validator-4.2.0.Final.jar hibernate-validator-annotation-processor-4.2.0.Final.jar slf4j-api-1.6.1.jar validation-api-1.0.0.GA.jar Once this is done, you simply annotate the 'Member' JavaBean we created earlier to indicate which properties need be validated. You can annotate either the fields or the accessor (or getter) methods of the JavaBean. In this post I will be annotating the accessor methods. package validationapiblog.model; import java.util.Date; import javax.validation.constraints.Max; import javax.validation.constraints.Min; import javax.validation.constraints.NotNull; import javax.validation.constraints.Past; import javax.validation.constraints.Pattern; import org.hibernate.validator.constraints.Email; import org.hibernate.validator.constraints.NotBlank; import validationapiblog.enums.Gender; /** * * @author Adedayo Ominiyi */ public class Member { private String lastName = null; private String firstName = null; private Gender gender = null; private String emailAddress = null; private Date dateOfBirth = null; public Member() { } @NotNull(message = "First name is compulsory") @NotBlank(message = "First name is compulsory") @Pattern(regexp = "[a-z-A-Z]*", message = "First name has invalid characters") public String getFirstName() { return firstName; } public void setFirstName(String firstName) { this.firstName = firstName; } @NotNull(message = "Gender is compulsory") public Gender getGender() { return gender; } public void setGender(Gender gender) { this.gender = gender; } @NotNull(message = "Last name is compulsory") @NotBlank(message = "Last name is compulsory") @Pattern(regexp = "[a-z-A-Z]*", message = "Last name has invalid characters") public String getLastName() { return lastName; } public void setLastName(String lastName) { this.lastName = lastName; } @Past(message = "Date of Birth must be the past") @NotNull public Date getDateOfBirth() { return dateOfBirth; } public void setDateOfBirth(Date dateOfBirth) { this.dateOfBirth = dateOfBirth; } @Min(value = 18, message = "Age must be greater than or equal to 18") @Max(value = 150, message = "Age must be less than or equal to 150") public Integer getAge() { if (this.dateOfBirth != null) { // calculate age of member here } return null; } @NotNull(message="Email Address is compulsory") @NotBlank(message="Email Address is compulsory") @Email(message = "Email Address is not a valid format") public String getEmailAddress() { return emailAddress; } public void setEmailAddress(String emailAddress) { this.emailAddress = emailAddress; } } Please note that these are just some of the annotations available in JSR 303. In addition Hibernate Validator introduces a few of its own that are not in the specification. Feel free to study the annotations not in this post in your free time you might find something interesting. There is also the ability to create your own custom validator if the need arises. Now lets review the annotations used: @NotNull - Checks that the annotated value is not null. Unfortunately it doesn't check for empty string values @Pattern - Checks if the annotated string matches the regular expression given. We used it to ensure that the last name and first name properties have valid string values @Past - The annotated element must be a date in the past. @Min - The annotated element must be a number whose value must be greater or equal to the specified minimum @Max - The annotated element must be a number whose value must be lower or equal to the specified maximum @NotBlank - Checks that the annotated string is not null and the trimmed length is greater than 0. This annotation is not in JSR 303 @Email - Checks whether the specified string is a valid email address. This annotation is also not in JSR 303 To test the validation we could use a unit test as shown below. package validationapiblog.test; import java.util.Set; import javax.validation.ConstraintViolation; import javax.validation.Validation; import javax.validation.Validator; import javax.validation.ValidatorFactory; import validationapiblog.model.Member; import org.junit.Test; import static org.junit.Assert.*; /** * * @author Adedayo Ominiyi */ public class ValidationAPIUnitTest { public ValidationAPIUnitTest() { } @Test public void testMemberWithNoValues() { Member member = new Member(); // validate the input ValidatorFactory factory = Validation.buildDefaultValidatorFactory(); Validator validator = factory.getValidator(); Set> violations = validator.validate(member); assertEquals(violations.size(), 5); } } In conclusion, you should experiment with JSR 303 and see for yourself which annotations you like. Thank you and have fun. Original URL: Bean Validation Made Simple With JSR 303

Looking for a "DevOps Manifesto" to help guide your own organizational transiton? Well, there's no 'manifesto' out there like we have for Agile and SOA, but I think if you look around, you'll find that this commuity does have some well-defined, agreed-upon goals. In Kris Buytaert's presentation: "Devops, the future is here, it's just not evenly distributed yet" he gives a great 10 point summary of organizational goals from Edward Deming that coincide with what the DevOps movment is looking to achieve in IT organizations: Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, and must learn their responsibilities, and take on leadership for a change. Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place. Constantly improve your system of production and service, to improve quality and productivity, and thus constantly decrease costs. Institute training on the job. Institute leadership. The aim of supervision should be to help people and machines and gadgets to a better job. Drive out fear, so that everyone may work effectively for the company. Break down barriers between departments. People in research, design, sales, and production must work as a team in order to forsee problems of production and usage that may be encountered with the product or service. Eliminate slogans, exhortation, and targets for the work force; asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, since the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force. *Eliminate management by objective. Eliminate management by numbers and numerical goals. Instead substitute with leadership. *Remove barriers that rob the hourly worker of his/her right to pride of workmanship. The responsibility of supervisors must be changed from sheer numbers to quality. *Remove barriers that rob people in management and in engineering of their right to pride of workmanship. Institute a vigorous program of education and self-improvement. Put everybody in the company to work to accomplish the transformation. The transformation is everybody's job. On a more technical level, Kris says you need the following things to get to a point where your team can deploy at a moment's notice, quickly and safely: Version Control Continuous Integration Built Pipelines Bug Tracking Integrated Testing Automated Deployment Here are all of the slides from his presentation: Devops, the future is here, it's just not evenly distributed yet. View more presentations from Kris Buytaert

// A fairly generic method to store arrays and also to add them to a pool to reverse lookup their ids based on values that they contain. This method extends my own redis client but will work for the better clients out there such as predis. class storage extends redis { public function save($key, array $object, $timestamp=true){ $timestamp && $object['timestamp'] = date('Ymdhis'); $id = $this->incr('id:'.$key); foreach ($object as $k => $v) { $this->sadd(sprintf("%s:%s:%s", $key, $k, $v), $id); } $key = sprintf("%s:%s", $key, $id); $this->hmset($key, $object); } }

In this post I will demonstrate how to leverage the -episode XJC extension to reuse classes previously generated from.an XML schema. This is useful when an XML schema is imported by other XML schemas and you do not want the same classes generated each time. Imported Schema (Product.xsd) The following XML schema represents basic information about a product. Product is a common concept in this example domain so I have decided to define one representation that can be leveraged by other schemas, rather than having each schema define its own representation of product information. Since multiple XML schemas import Product.xsd we can leverage episode files so that the classes corresponding to Product.xsd are only generated once. The following XJC call demonstrates how to generate an episode file called product.episode along with the generated classes: xjc -d out -episode product.episode Product.xsd Importing Schema (ProductPurchaseRequest.xsd) Below is an example of an XML schema that imports Product.xsd: When we generate classes from this XML schema we will reference the episode file we created when we generated Java classes from Product.xsd. If we do not specify the episode file then classes will be generated for both ProductPurchaseRequest.xsd and Product.xsd: xjc -d out ProductPurchaseRequest.xsd -extension -b product.episode Another Importing Schema (ProductQuoteRequest.xsd) Below is another example of an XML schema that imports Product.xsd: Again when we generate classes from this XML schema we will reference the episode file we created when we generated Java classes from Product.xsd. xjc -d out ProductQuoteRequest.xsd -extension -b product.episode How Does it Work? (product.episode) For those of you curious how this works. The episode file generated by XJC is really just a standard JAXB bindings file that is used to customize the class generation. This generated bindings/episode file contains entries that tells XJC that a class already exists for this type. You could write this file by hand, but -episode flag to XJC does it for you. From http://blog.bdoughan.com/2011/12/reusing-generated-jaxb-classes.html

This post examines what options one has for handling footnotes and references in HTML. It then presents a library that helps you with handling them. Requirements Handling footnotes and references comes with several requirements: On screen, one wants to show the footnote text as close as possible to the number pointing to the footnote. Whatever solution one chooses, it should also work on touch devices. Hence, a hover-only approach is not feasible. In print, footnotes should be shown, as well. Hence, a tooltip-only solution is not acceptable. Lastly, things should degrade gracefully if JavaScript is switched off. The HTML5 spec recommendations for footnotes The HTML5 specification gives several tips on how to format footnotes. Short inline annotations: title attribute. Customer: Hello! I wish to register a complaint. Hello. Miss? Shopkeeper: Watcha mean, miss? Customer: Uh, I'm sorry, I have a cold. I wish to make a complaint. Shopkeeper: Sorry, we're closing for lunch. Longer annotations: bidirectional linking via : Announcer: Number 16: The hand. Interviewer: Good evening. I have with me in the studio tonight Mr Norman St John Polevaulter, who for the past few years has been contradicting people. Mr Polevaulter, why do you contradict people? Norman: I don't. [1] Interviewer: You told me you did! ... [1] This is, naturally, a lie, but paradoxically if it were true he could not say so without contradicting the interviewer and thus making it false. Wikipedia-style highlighting of the currently active footnote The CSS pseudo-selector :target allows you to style the HTML element whose ID is the same as the page fragment identifier: li:target { background-color: #BFEFFF; } Thus, if the page URL ends with #explanation then the following list item would be highlighted: It works like this: ... Using the html_footnotes library You can download html_footnotes on GitHub and try it out online. Terminology: An annotation is either a footnote or a reference (citation). The markers in the main text referring to those annotations are called annotation pointers. A footnote pointer is a number written in parentheses. Example: (1) A reference pointer is a number written in square brackets. Example: [1] Activating the library: The library consists of CSS to style annotations and pointers and of JavaScript that post-processes the HTML so that less code has to be written. Footnotes: The library processes the HTML so that footnote pointers are formatted as superscript and become links that, when clicked on, display the text of the footnote inline. You can use an IIFE(1) to avoid the global namespace(2) being polluted. ... Footnotes IIFE is an acronym for Immediately-Invoked Function Expression.The global scope is reified as an object in JavaScript. References: Reference pointers are processed and become links. The library also adds IDs to the reference list items. Due to the appropriate CSS, a list item will be highlighted and scrolled to if one clicks on a pointer. JavaScript has many functional language constructs [1]. For example: consult [2] for an introduction to closures. ... References Functional programming. In Wikipedia. Retrieved 2011-12-03.Douglas Crockford, JavaScript: The Good Parts. O’Reilly. 2008-05-16. Source: http://www.2ality.com/2011/12/footnotes.html

this post presents an analysis of mysql (a relational database) and neo4j (a graph database) in a side-by-side comparison on a simple graph traversal. the data set that was used was an artificially generated graph with natural statistics. the graph has 1 million vertices and 4 million edges. the degree distribution of this graph on a log-log plot is provided below. a visualization of a 1,000 vertex subset of the graph is diagrammed above. loading the graph the graph data set was loaded both into mysql and neo4j. in mysql a single table was used with the following schema. create table graph ( outv int not null, inv int not null ); create index outv_index using btree on graph (outv); create index inv_index using btree on graph (inv); after loading the data, the table appears as below. the first line reads: “vertex 0 is connected to vertex 1.” mysql> select * from graph limit 10; +------+-----+ | outv | inv | +------+-----+ | 0 | 1 | | 0 | 2 | | 0 | 6 | | 0 | 7 | | 0 | 8 | | 0 | 9 | | 0 | 10 | | 0 | 12 | | 0 | 19 | | 0 | 25 | +------+-----+ 10 rows in set (0.04 sec) the 1 million vertex graph data set was also loaded into neo4j. in gremlin , the graph edges appear as below. the first line reads: “vertex 0 is connected to vertex 992915.” gremlin> g.e[1..10] ==>e[183][0-related->992915] ==>e[182][0-related->952836] ==>e[181][0-related->910150] ==>e[180][0-related->897901] ==>e[179][0-related->871349] ==>e[178][0-related->857804] ==>e[177][0-related->798969] ==>e[176][0-related->773168] ==>e[175][0-related->725516] ==>e[174][0-related->700292] warming up the caches before traversing the graph data structure in both mysql and neo4j, each database had a “ warm up ” procedure run on it. in mysql, a “select * from graph” was evaluated and all of the results were iterated through. in neo4j, every vertex in the graph was iterated through and the outgoing edges of each vertex were retrieved. finally, for both mysql and neo4j, the experiment discussed next was run twice in a row and the results of the second run were evaluated. traversing the graph the traversal that was evaluated on each database started from some root vertex and emanated n-steps out. there was no sorting, no distinct-ing, etc. the only two variables for the experiments are the length of the traversal and the root vertex to start the traversal from. in mysql, the following 5 queries denote traversals of length 1 through 5. note that the “?” is a variable parameter of the query that denotes the root vertex. select a.inv from graph as a where a.outv=? select b.inv from graph as a, graph as b where a.inv=b.outv and a.outv=? select c.inv from graph as a, graph as b, graph as c where a.inv=b.outv and b.inv=c.outv and a.outv=? select d.inv from graph as a, graph as b, graph as c, graph as d where a.inv=b.outv and b.inv=c.outv and c.inv=d.outv and a.outv=? select e.inv from graph as a, graph as b, graph as c, graph as d, graph as e where a.inv=b.outv and b.inv=c.outv and c.inv=d.outv and d.inv=e.outv and a.outv=? for neo4j, the blueprints pipes framework was used. a pipe of length n was constructed using the following static method. public static pipeline createpipeline(final integer steps) { final arraylist pipes = new arraylist(); for (int i = 0; i < steps; i++) { pipe pipe1 = new vertexedgepipe(vertexedgepipe.step.out_edges); pipe pipe2 = new edgevertexpipe(edgevertexpipe.step.in_vertex); pipes.add(pipe1); pipes.add(pipe2); } return new pipeline(pipes); } for both mysql and neo4j, the results of the query (sql and pipes) were iterated through. thus, all results were retrieved for each query. in mysql, this was done as follows. while (resultset.next()) { resultset.getint(finalcolumn); } in neo4j, this is done as follows. while (pipeline.hasnext()) { pipeline.next(); } experimental results the artificial graph dataset was constructed with a “ rich get richer “, preferential attachment model . thus, the vertices created earlier are the most dense (i.e. highest number of adjacent vertices). this property was used to limit the amount of time it would take to evaluate the tests for each traversal. only the first 250 vertices were used as roots of the traversals. before presenting timing results, note that all of these experiments were run on a macbook pro with a 2.66ghz intel core 2 duo and 4gigs of ram at 1067 mhz ddr3. the packages used were java 1.6, mysql jdbc 5.0.8, and blueprints pipes 0.1.2. java version "1.6.0_17" java(tm) se runtime environment (build 1.6.0_17-b04-248-10m3025) java hotspot(tm) 64-bit server vm (build 14.3-b01-101, mixed mode) the following java virtual machine parameters were used: -xmx1000m -xms500m below are the total running times for both mysql (red) and neo4j (blue) for traversals of length 1, 2, 3, and 4. the raw data is presented below along with the total number of vertices returned by each traversal—which, of course, is the same for both mysql and neo4j given that its the same graph data set being processed. also realize that traversals can loop and thus, many of the same vertices are returned multiple times. finally, note that only neo4j has the running time for a traversal of length 5. mysql did not finish after waiting 2 hours to complete. in comparison, neo4j took 14.37 minutes to complete a 5 step traversal. [mysql steps-1] time(ms):124 -- vertices_returned:11360 [mysql steps-2] time(ms):922 -- vertices_returned:162640 [mysql steps-3] time(ms):8851 -- vertices_returned:2206437 [mysql steps-4] time(ms):112930 -- vertices_returned:28125623 [mysql steps-5] n/a [neo4j steps-1] time(ms):27 -- vertices_returned:11360 [neo4j steps-2] time(ms):474 -- vertices_returned:162640 [neo4j steps-3] time(ms):3366 -- vertices_returned:2206437 [neo4j steps-4] time(ms):49312 -- vertices_returned:28125623 [neo4j steps-5] time(ms):862399 -- vertices_returned:358765631 next, the individual data points for both mysql and neo4j are presented in the plot below. each point denotes how long it took to return n number of vertices for the varying traversal lengths. finally, the data below provides the number of vertices returned per millisecond (on average) for each of the traversals. again, mysql did not finish in its 2 hour limit for a traversal of length 5. [mysql steps-1] vertices/ms:91.6128847554668 [mysql steps-2] vertices/ms:176.399127537985 [mysql steps-3] vertices/ms:249.286746556076 [mysql steps-4] vertices/ms:249.053599519823 [mysql steps-5] n/a [neo4j steps-1] vertices/ms:420.740351166341 [neo4j steps-2] vertices/ms:343.122344772028 [neo4j steps-3] vertices/ms:655.507125256186 [neo4j steps-4] vertices/ms:570.360621871775 [neo4j steps-5] vertices/ms:416.00886711325 conclusion in conclusion, given a traversal of an artificial graph with natural statistics, the graph database neo4j is more optimal than the relational database mysql. however, no attempts have been made to optimize the java vm, the sql queries, etc. these experiments were run with both neo4j and mysql “out of the box” and with a “natural syntax” for both types of queries. source: http://markorodriguez.com/2011/02/18/mysql-vs-neo4j-on-a-large-scale-graph-traversal/

The aim of this article is to learn how to dynamically trace/observe running Java applications (JDK 6+) using BTrace without changing the code and configuration params of the applications. What is BTrace? BTrace is an open source project that was started in 2007 and was originally owned by two people – A.Sundararajan and K. Balasubramanian. It gained fame thanks to Java One 2008 conference. BTrace helps us to pinpoint complicated code problems in the application. The problems that can be solved using BTrace include code bugs, unpredictable flows, concurrency problems and performance problems that happen under specific and usually hard to reproduce circumstances. BTrace dynamically (without restarting application) instruments (changes) bytecode of an application in the way that programmer defines. The goal of the instrumentation is to see what happens in a specific area of the code. If used beyond this scope it may harm applications' flow and therefore is forbidden by a validator. For example let's try to solve following problem – an important file gets deleted occasionally once a day. We want to find the code that does this. Therefore we would like to change "delete" method of java.io.File and print a stack trace of the calling thread if the file name fits. With BTrace we can do this by writing a short and straightforward Java code. The following schema displays principles of BTrace work. In Target JVM there is a dynamically inserted BTrace agent (using attach API). BTrace client (either BTrace Command Line or Visual VM with BTrace plugin) sends commands to the agent and gets responses. BTrace agent instruments classes that are being loaded into Target JVM and reloads classes that have already been loaded. Writing a Tracing script Writing BTrace scripts is pretty simple and straightforward and has concepts similar to Aspect Oriented Programming concepts. import com.sun.btrace.annotations.*; import com.sun.btrace.BTraceUtils; @BTrace public class HelloWorld { @OnMethod(clazz="java.io.File",method="") public static void onNewFileCreated(String fileName) { BTraceUtils.println("New file is being created"); BTraceUtils.println(fileName); } Each BTrace script consists of Probes (pointcuts in Aspects slang) and Actions (advices). A probe defines when the instrumentation should be executed. Action defines the instrumentation. Probe can consist of the following: Method entry/exit Line number Field updated/accessed Method call/return (within specified method(s)) Exception throw (before) Synchronization entry/exit Timer Constructor entry Abilities and limitations of BTrace Abilities Dynamically connect to any java6 + application and run any(*) code Limitations Must run with the same user as the traced application is running with - due to security concerns Supports Hotspot JVM only Is compiled separately from the target application – is not familiar with application classes May crush target application - I ran BTrace on our longevity setup which was heavily loaded with many Tracing scripts. Target application didn't crush even once. Advanced BTrace BTrace community consists de facto of a single developer that works regularly on the project. Therefore don't expect too much improvement and many new features in the next releases. (This is a good opportunity to contribute to the project as each developer will enhance the development significantly). Documentation of the project should be significantly improved - I found myself guessing many times while integrating this framework into the existing frameworks at my job. BTrace forum is a good way to get answers about the framework. In order to understand how the magic of BTrace works, knowledge in three fields is required - "Java Agents Development", "Bytecode manipulation in java" and "Java attach api". I may write about some of these fields in the future. References Check out my blog http://www.parleys.com/#st=5&id=1618&sl=1 http://kenai.com/projects/btrace

In a previous post I covered how to use namespace qualification with JAXB. In this post I will cover how to control the prefixes that are used. This is not covered in the JAXB (JSR-222) specification but I will demonstrate the extensions available in both the reference and EclipseLink MOXy implementations for handling this use case Java Model The following domain model will be used for this post. The @XmlRootElement and @XmlElement annotation are used to specify the appropriate namespace qualification. package blog.prefix; import javax.xml.bind.annotation.XmlElement; import javax.xml.bind.annotation.XmlRootElement; @XmlRootElement(namespace="http://www.example.com/FOO") public class Root { private String a; private String b; private String c; @XmlElement(namespace="http://www.example.com/BAR") public String getA() { return a; } public void setA(String a) { this.a = a; } @XmlElement(namespace="http://www.example.com/FOO") public String getB() { return b; } public void setB(String b) { this.b = b; } @XmlElement(namespace="http://www.example.com/OTHER") public String getC() { return c; } public void setC(String c) { this.c = c; } } Demo Code We will use the following code to populate the domain model and produce the XML. package blog.prefix; import javax.xml.bind.JAXBContext; import javax.xml.bind.Marshaller; public class Demo { public static void main(String[] args) throws Exception { JAXBContext ctx = JAXBContext.newInstance(Root.class); Root root = new Root(); root.setA("A"); root.setB("B"); root.setC("OTHER"); Marshaller m = ctx.createMarshaller(); m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true); m.marshal(root, System.out); } } Output XML like the following is produced by default. The JAXB implementation has arbitrarily assigned prefixes to the namespace URIs specified in the domain model: A B OTHER Specify Prefix Mappings with JAXB RI & Metro JAXB The reference and Metro implementations of JAXB provide a mechanism called NamespacePrefixMapper to control the prefixes that will be assigned to namespaces. NamespacePrefixMapper package blog.prefix; import com.sun.xml.internal.bind.marshaller.NamespacePrefixMapper; //import com.sun.xml.bind.marshaller.NamespacePrefixMapper; public class MyNamespaceMapper extends NamespacePrefixMapper { private static final String FOO_PREFIX = ""; // DEFAULT NAMESPACE private static final String FOO_URI = "http://www.example.com/FOO"; private static final String BAR_PREFIX = "bar"; private static final String BAR_URI = "http://www.example.com/BAR"; @Override public String getPreferredPrefix(String namespaceUri, String suggestion, boolean requirePrefix) { if(FOO_URI.equals(namespaceUri)) { return FOO_PREFIX; } else if(BAR_URI.equals(namespaceUri)) { return BAR_PREFIX; } return suggestion; } @Override public String[] getPreDeclaredNamespaceUris() { return new String[] { FOO_URI, BAR_URI }; } } Demo Code The NamespacePrefixMapper is set on an instance of Marshaller. I would recommend wrapping the setPropery call in a try/catch block so that your application does not fail if you change JAXB implementations. package blog.prefix; import javax.xml.bind.JAXBContext; import javax.xml.bind.Marshaller; public class Demo { public static void main(String[] args) throws Exception { JAXBContext ctx = JAXBContext.newInstance(Root.class); Root root = new Root(); root.setA("A"); root.setB("B"); root.setC("OTHER"); Marshaller m = ctx.createMarshaller(); m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true); try { m.setProperty("com.sun.xml.internal.bind.namespacePrefixMapper", new MyNamespaceMapper()); //m.setProperty("com.sun.xml.bind.namespacePrefixMapper", new MyNamespaceMapper()); } catch(PropertyException e) { // In case another JAXB implementation is used } m.marshal(root, System.out); } } Output The resulting document now uses the NamespacePrefixMapper to determine the prefixes that should be used in the resulting document. A B OTHER Specify Prefix Mappings with EclipseLink JAXB (MOXy) MOXy will use the namespace prefixes as they are defined on the @XmlSchema annotation. In order for MOXy to be able to use the default namespace the elementFormDefault property on the @XmlSchema annotation must be set to XmlNsForm.QUALIFIED. package-info XmlSchema( elementFormDefault=XmlNsForm.QUALIFIED, namespace="http://www.example.com/FOO", xmlns={@XmlNs(prefix="bar", namespaceURI="http://www.example.com/BAR")} ) package blog.prefix; import javax.xml.bind.annotation.XmlNs; import javax.xml.bind.annotation.XmlNsForm; import javax.xml.bind.annotation.XmlSchema; Output The resulting document now uses the xmlns setting from the @XmlSchema annotation to determine the prefixes that should be used in the resulting document. A B OTHER Further Reading If you enjoyed this post then you may also be interested in: JAXB & Namespaces Specifying EclipseLink MOXy as Your JAXB Provider From http://blog.bdoughan.com/2011/11/jaxb-and-namespace-prefixes.html

Develop your own API service for your PHP projects.

In the scenario of today, two or more parameters are often passed together to a set of similar methods. This happens for example with timing information containing day and month, or hours and minutes; or, in other domains, with parameters that are coupled to each other - such as an host and a port (google.com, 80), or an image and its alt text. A long list of coupled parameters is not necessarily the sign that a method does too much. Today's refactoring is a solution to simplify the signature and express the coupling between the current parameters: wrapping them into a Parameter Object. Why writing code for a new class? We shouldn't be afraid of adding classes, at the similar pace as we add methods. When parameters are almost always passed together, they will be written down together in each signature and each call: it's a form of duplication and as such can and should be eliminated. Moreover, the duplication is not limited to a single method: the set of coupled arguments can be passed to multiple methods, each called in many more places which must be synchronized. For example, adding a parameter in the initial situation means modifying lots of different source files. The parameters are often the sign of an hidden concept: an object modelling their union. You know you are on the right track if a name for this concepts comes up quickly; otherwise you will have to invent it. This refactoring enables you to put more logic on an object representing this set of values, instead of passing them around in an array (Primitive Obsession) or, like in this case, always together as multiple parameters. The end result is also a shorter and more clear parameter list, again reaching the goal of simplifying method calls as we are doing with lots of refactorings in this part of the series. Steps Create a new class: an instance of this class should wrap the various values which are passed to it in the constructor. The Parameter Object will usually be a Value Object, with an immutable state and with getters for each of its fields (for now). Execute Add Parameter to pass in also the new object (created on the spot) together with the various parameters. For each of the old parameters, apply Remove Parameter on it and access it inside the method by calling a getter on the Parameter Object. The tests should always pass between the steps. The final step enabled by this refactoring consists in moving logic onto the Parameter Object. If you're lucky, this even leads to some of the getters vanishing. Example We have an Invoice object, which accepts some rows in order to calculate a total. A copious amount of money covering the Value Added Tax is added to the net total. addRow(1000); $quotation->specifyTaxes(20, 'VATCODE'); $this->assertEquals(1200, $quotation->getTotal()); $this->assertEquals('Type: VATCODE', $quotation->getTypeOfService()); } } class Quotation { private $netTotal = 0; private $vatPercentage; private $vatCode; public function addRow($row) { // including just the logic to implement the test we have $this->netTotal += $row; } public function specifyTaxes($percentage, $code) { $this->vatPercentage = $percentage; $this->vatCode = $code; } public function getTotal() { return $this->netTotal * (1 + $this->vatPercentage / 100); } public function getTypeOfService() { return 'Type: ' . $this->vatCode; } } We notice two things: one in the rest of the code (not shown here): the 20% percentage and its related code are passed together to many methods. It makes sense, as different codes (bread vs. cars) may imply different tax percentages. in the class itself, the fields storing tax informations have very similar names: $vatRate and $vatCode. This is a mild form of duplication. So we want to transform the couple of parameters into a Parameter Object. We create the class we need: class VatRate { private $rate; private $code; public function __construct($rate, $code) { $this->rate = $rate; $this->code = $code; } public function getRate() { return $this->rate; } public function getCode() { return $this->code; } } Now we can add a parameter to the method, which is an instance of our Parameter Object: addRow(1000); $quotation->specifyTaxes(new VatRate(20, 'VATCODE'), 20, 'VATCODE'); $this->assertEquals(1200, $quotation->getTotal()); $this->assertEquals('Type: VATCODE', $quotation->getTypeOfService()); } } class Quotation { private $netTotal = 0; private $vatRate; private $vatPercentage; private $vatCode; public function addRow($row) { // including just the logic to implement the test we have $this->netTotal += $row; } public function specifyTaxes(VatRate $vatRate, $percentage, $code) { $this->vatRate = $vatRate; $this->vatPercentage = $percentage; $this->vatCode = $code; } public function getTotal() { return $this->netTotal * (1 + $this->vatPercentage / 100); } public function getTypeOfService() { return 'Type: ' . $this->vatCode; } } Invoice objects have all the information they need, available from the VatRate instance. So let's remove $percentage, the first parameter wrapped into the Parameter Object: class Quotation { private $netTotal = 0; private $vatRate; private $vatPercentage; private $vatCode; public function addRow($row) { // including just the logic to implement the test we have $this->netTotal += $row; } public function specifyTaxes(VatRate $vatRate, $code) { $this->vatRate = $vatRate; $this->vatCode = $code; } public function getTotal() { return $this->netTotal * (1 + $this->vatRate->getRate() / 100); } public function getTypeOfService() { return 'Type: ' . $this->vatCode; } } And now we can also remove $code, the other parameter: class Quotation { private $netTotal = 0; private $vatRate; public function addRow($row) { // including just the logic to implement the test we have $this->netTotal += $row; } public function specifyTaxes(VatRate $vatRate) { $this->vatRate = $vatRate; } public function getTotal() { return $this->netTotal * (1 + $this->vatRate->getRate() / 100); } public function getTypeOfService() { return 'Type: ' . $this->vatRate->getCode(); } } We notice this refactoring has enabled a further step: moving the calculation of the tax into VatRate. This also means we can delete the getRate() method. class Quotation { private $netTotal = 0; private $vatRate; public function addRow($row) { // including just the logic to implement the test we have $this->netTotal += $row; } public function specifyTaxes(VatRate $vatRate) { $this->vatRate = $vatRate; } public function getTotal() { return $this->vatRate->tax($this->netTotal); } public function getTypeOfService() { return 'Type: ' . $this->vatRate->getCode(); } } class VatRate { private $rate; private $code; public function __construct($rate, $code) { $this->rate = $rate; $this->code = $code; } public function getCode() { return $this->code; } public function tax($netAmount) { return $netAmount * (1 + $this->rate / 100); } }

Guava currently (Release 10) includes four annotations in its com.google.common.annotations package: Beta, VisibleForTesting, GwtCompatible, and GwtIncompatible. The last two are specific to use with Google Web Toolkit (GWT), but the former two can be useful in a more general context. The @Beta annotation is used within Guava's own code base to indicate "that a public API (public class, method or field) is subject to incompatible changes, or even removal, in a future release." Although this annotation is used to indicate at-risk public API constructs in Guava, it can also be used in code that has access to Guava on its classpath. Developers can use this annotation to advertise their own at-risk public API constructs. The @Beta annotation is defined as a @Documented, which means that it marks something that is part of the public API and should be considered by Javadoc and other source code documentation tools. The @VisibleForTesting annotation "indicates that the visibility of a type or member has been relaxed to make the code testable." I have never liked having to relax type or member visibility to make something testable. It feels wrong to have to compromise one's design to allow testing to occur. This annotation is better than nothing in such a case because it at least makes it clear to others using the construct that there is a reason for its otherwise surprisingly relaxed visibility. Conclusion Guava provides two annotations that are not part of the standard Java distribution, but cover situations that we often run into during Java development. The @Beta annotation indicates a construct in a public API that may be changed or removed. The @VisibleForTesting annotation advertises to other developers (or reminds the code's author) when a decision was made for relaxed visibility to make testing possible or easier. From http://marxsoftware.blogspot.com/2011/11/two-generally-useful-guava-annotations.html

For most large web applications, uptime is of foremost importants. Any outage can be seen by customers as a frustration, or opportunity to move to a competitor. What's more for a site that also includes e-commerce, it can mean real lost sales. Zero Downtime describes a site without service interruption. To achieve such lofty goals, redundancy becomes a critical requirement at every level of your infrastructure. If you're using cloud hosting, are you redundant to alternate availability zones and regions? Are you using geographically distributed load balancing? Do you have multiple clustered databases on the backend, and multiple webservers load balanced. All of these requirements will increase uptime, but may not bring you close to zero downtime. For that you'll need thorough testing. The solution is to pull the trigger on sections of your infrastructure, and prove that it fails over quickly without noticeable outage. The ultimate test is the outage itself. Sean Hull on Quora: What is zero downtime and why is it important? Source: http://www.iheavy.com/2011/06/23/zero-downtime-what-is-it-and-why-is-it-important/

One of NoSQL's goals: handle previously-unthinkable amounts of data. One of unthinkable-amounts-of-data's problems: previously-improbable events become extremely probable, precisely because the set of interactions is so large. Flip a coin a hundred times, and you're not likely to get 50 heads in a row. But flip it a few trillion times, and you probably will find some 50-heads streaks. So NoSQL's performance strength is also its mathematical weakness. This order of scale can result in lots of problems, but one of the most common is consistency -- the C in ACID -- clearly a fundamental desideratum for any database system, but in principle much harder to acheive for NoSQL databases than for others. Emerging database technologies have forced developers and computer scientists to define more exactly what kind of consistency is really needed, for any given application. Two years ago, ACM (the Association for Computing Machinery) published an extremely helpful examination of the attenuated notion of consistency called 'eventual consistency'. Their summary: Data inconsistency in large-scale reliable distributed systems must be tolerated for two reasons: improving read and write performance under highly concurrent conditions; and handling partition cases where a majority model would render part of the system unavailable even though the nodes are up and running. The article surveys technical solutions as well as user considerations that might soften the undesirability of anything less than perfect, instantaneous consistency. It's not long (4 pages plus pictures), and explains some deep database issues quite clearly. On the more practical side of the problem: Russell Brown recently gave a talk at the NoSQL Exchange 2011 on exactly this topic. More specifically, he showed how some distributed systems (Riak in particular) try to minimize conflicts, and suggested some ways to reconcile conflicts automatically using smart semantic techniques. Check out the NoSQL Exchange page for Russell's talk here, which includes an embedded video. But read the ACM article first for a broader overview, since Russell launches into technical details pretty quickly.

Most databases have them: a small handful of big tables that grow at a substantially faster rate than any of their peers (think tweets, clicks, check-ins, etc). Over time, the sheer size of these tables cause queries against them to slow down, and increase the size and time taken for backups. In some cases, not all of this data needs to be “live” so that it can be randomly accessed forever, and can be archived once some criteria has been met. I like to think of this as Eventually Irrelevant (if I may coin a term). This post was authored by Brian Ploetz Many database products support partitioning features, which allow you to arrange data physically based on some criteria (typically date, range, list, or hash-based partitioning). These partitioning features allow you to drop partitions from the live database when appropriate. While MongoDB supports horizontal partitioning via sharding, it does not currently support the notion of dropping shards. Even if it did, the criteria for what documents to drop from a collection would most likely not be based on your shard key, so it’s unlikely you’d want to drop a shard anyways. You’d want to drop data from all shards. Capped Collections and the forthcoming TTL-based Capped Collections feature are not quite what we’re looking for either. Capped Collections have very strict rules (including not being able to shard them), and for both normal and TTL-based Capped Collections, as old documents are aged out they are simply dropped on the floor. What we need is more control over the dropping process, such that we can guarantee that our data has been copied to an archive database/data warehouse before being dropped from the main database. I set out to find the most efficient way of manually archiving documents from a large collection given the tools currently available in MongoDB. For the purposes of this exercise, let’s assume we have a large collection of ad clicks which are associated with ads. We want to archive clicks which are associated with ads which expired more than 3 months ago. MapReduce MongoDB wants you to do bulk operations using MapReduce. Since all of the examples and documentation revolve around aggregation, I suspected I wouldn’t be able to use MapReduce as the backbone of my archiving process. After some experimenting, I found that my intuition was correct. Upon attempting to insert/remove documents within the map or reduce functions, I would quickly run into exception 10293 internal error: locks are not upgradeable errors: Sun Sep 25 21:22:24 [conn5] update warehouse.clicks query: { _id: ObjectId('4d6bf191db0b4b2b1fad5b65') } exception 10293 internal error: locks are not upgradeable: { "opid" : 4248173, "active" : false, "waitingForLock" : false, "op" : "update", "ns" : "?", "query" : { "_id" : { "$oid" : "4d6bf191db0b4b2b1fad5b65" } }, "client" : "0.0.0.0:0", "desc" : "conn" } 0ms Sun Sep 25 21:22:24 [conn5] Assertion: 10293:internal error: locks are not upgradeable: { "opid" : 4248174, "active" : false, "waitingForLock" : false, "op" : "update", "ns" : "?", "query" : { "_id" : { "$oid" : "4d6bf198db0b4b2b20ad5b65" } }, "client" : "0.0.0.0:0", "desc" : "conn" } 0x10008de9b 0x1002064f7 0x1002c1ec6 0x1002c4f3d 0x1002c5f7a 0x1000a8181 0x100147df4 0x1004dc68e 0x1004efc93 0x1004dc71b 0x1004dcb71 0x10049a078 0x100158efa 0x100377760 0x100389d0c 0x10034c204 0x10034d877 0x100180cc4 0x100184649 0x1002b9e89 0 mongod 0x000000010008de9b _ZN5mongo11msgassertedEiPKc + 315 1 mongod 0x00000001002064f7 _ZN5mongo10MongoMutex19_writeLockedAlreadyEv + 263 2 mongod 0x00000001002c1ec6 _ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE + 886 3 mongod 0x00000001002c4f3d _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE + 5661 4 mongod 0x00000001002c5f7a _ZN5mongo14DBDirectClient3sayERNS_7MessageE + 106 5 mongod 0x00000001000a8181 _ZN5mongo12DBClientBase6updateERKSsNS_5QueryENS_7BSONObjEbb + 273 6 mongod 0x0000000100147df4 _ZN5mongo12mongo_updateEP9JSContextP8JSObjectjPlS4_ + 660 7 mongod 0x00000001004dc68e js_Invoke + 3864 8 mongod 0x00000001004efc93 js_Interpret + 71932 9 mongod 0x00000001004dc71b js_Invoke + 4005 10 mongod 0x00000001004dcb71 js_InternalInvoke + 404 11 mongod 0x000000010049a078 JS_CallFunction + 86 12 mongod 0x0000000100158efa _ZN5mongo7SMScope6invokeEyRKNS_7BSONObjEib + 666 13 mongod 0x0000000100377760 _ZN5mongo2mr8JSMapper3mapERKNS_7BSONObjE + 96 14 mongod 0x0000000100389d0c _ZN5mongo2mr16MapReduceCommand3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb + 1740 15 mongod 0x000000010034c204 _ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 628 16 mongod 0x000000010034d877 _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi + 2151 17 mongod 0x0000000100180cc4 _ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi + 52 18 mongod 0x0000000100184649 _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 10585 19 mongod 0x00000001002b9e89 _ZN5mongo13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 569 Upon killing the client which kicked off the MapReduce process, theses errors would continue to spin out of control, and I’d have to bounce the server. While MapReduce’s divide and conquer approach is ideal for what we’re trying to accomplish, unless the ability to obtain a write lock within a MapReduce function is supported, it is not a viable option for now. db.eval() While a server side script is certainly a viable option for writing an archiving process, it has a fundamental flaw in that it won’t work for sharded collections, which could be a non-starter for some folks. An example archiving process using db.eval() would look something like this: archiveClick = function archiveClick(doc) { if (expiredAdIds.indexOf(doc.ad_id) != -1) { archivedb = db.getMongo().getDB("archive"); archivedb.clicks.save(doc); // before we remove the original document, make sure the archive worked if (archivedb.getLastError() == null) { db.clicks.remove({_id: doc._id}); } else { throw "could not archive document with _id " + doc._id; } } } archiveClicks = function archiveClicks() { threeMonthsAgo = new Date(); threeMonthsAgo.setMonth(threeMonthsAgo.getMonth()-3); expiredAdIds = []; getExpiredAds = function(ad) {expiredAdIds.push(ad._id.str);} db.ads.find({end_date: {$lte: threeMonthsAgo}).forEach(getExpiredAds); db.system.js.save({_id: "expiredAdIds", value: expiredAdIds}); db.clicks.find().forEach(archiveClick); db.system.js.remove({_id: "expiredAdIds"}); } db.system.js.save({_id: "archiveClick", value: archiveClick}); db.system.js.save({_id: "archiveClicks", value: archiveClicks}); db.runCommand({$eval: "archiveClicks()", nolock: true}); Note the nolock: true sent to db.eval(). This ensures the mongod isn’t locked for other operations while this runs. The Holy Grail: Define Custom archive() Functions On Collections In order for our archiving process to work for sharded collections, we will need to add new functions to collection objects in the shell to perform the archiving. Frankly, this feels a lot more natural than db.eval(). I wanted to support two modes of archiving: immediate archiving, and a mark & sweep approach which allows you to queue documents to be archived at one point in time, and perform the actual archiving at a later time (off-peak hours). I’ve made these functions available in my mongodb-archive project on GitHub. Download the archive.js file, and use it like so: load("archive.js"); threeMonthsAgo = new Date(); threeMonthsAgo.setMonth(threeMonthsAgo.getMonth()-3); expiredAdIds = []; getExpiredAds = function(ad) {expiredAdIds.push(ad._id.str);} db.ads.find({end_date: {$lte: threeMonthsAgo}).forEach(getExpiredAds); archiveConnection = new Mongo("localhost:27018"); archiveDB = archiveConnection.getDB("archive"); archiveCollection = archiveDB.getCollection("clicks"); for (var i = 0; i < expiredAdIds.length; i++) { print("archiving clicks for ad_id: " + expiredAdIds[i]); db.clicks.archive({"ad_id": expiredAdIds[i]}, archiveCollection); print(""); } The mark/sweep variant would look like: db.clicks.queueForArchive({"ad_id": expiredAdIds[i]}); // ....some time later..... db.clicks.archiveQueued(archiveCollection); The biggest factors on the performance of the archiving process are really no different than the things that impact the performance of your MongoDB database overall: working set size, indexes, disk speed, and lock contention. The more indexes there are on the source collection, the longer the remove() will take. The more indexes there are on the archive collection, the longer the save() will take. The more of the data that you wish to archive is resident in memory, the faster the process will be. On my MacBook Pro (2.3GHz i7, 8GB RAM) with a 5400 RPM SATA drive and a clicks collection containing over 11 million documents, I was able to archive ~500 documents per second with the immediate archiving approach. Example output from mongostat when this was running: Archive DB: insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 1 0 0 3 0 208m 2.85g 15m 3.8 0 0|0 0|0 1k 1k 2 17:13:02 0 0 45 0 0 46 0 208m 2.85g 15m 3.4 0 0|0 0|0 61k 6k 2 17:13:03 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 8 0 0 9 0 208m 2.85g 15m 0.1 0 0|0 0|0 10k 2k 2 17:13:04 0 0 47 0 0 48 0 208m 2.85g 15m 3.4 0 0|0 0|0 66k 7k 2 17:13:05 0 0 0 0 0 1 1 208m 2.85g 15m 0 0 0|0 0|0 62b 1k 2 17:13:06 0 0 18 0 0 19 0 208m 2.85g 15m 2.6 0 0|0 0|0 24k 3k 2 17:13:07 0 0 79 0 0 80 0 208m 2.85g 16m 1.1 0 0|0 0|0 111k 11k 2 17:13:08 0 0 131 0 0 132 0 208m 2.85g 16m 8.8 0 0|0 0|0 181k 17k 2 17:13:09 0 0 216 0 0 217 0 208m 2.85g 16m 3.8 0 0|0 0|0 291k 27k 2 17:13:10 0 0 425 0 0 426 0 208m 2.85g 17m 10.5 0 0|0 0|0 600k 53k 2 17:13:11 0 0 490 0 0 490 0 208m 2.85g 19m 7.3 0 0|0 0|1 658k 61k 2 17:13:12 0 0 630 0 0 632 0 208m 2.85g 20m 11.3 0 0|0 0|0 844k 78k 2 17:13:13 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 818 0 0 818 0 208m 2.85g 22m 13.6 0 0|0 0|0 1m 101k 2 17:13:14 0 0 686 0 0 688 0 208m 2.85g 21m 9.6 0 0|0 0|0 932k 85k 2 17:13:15 0 0 0 0 0 1 0 208m 2.85g 22m 0 0 0|0 0|0 62b 1k 2 17:13:16 0 0 178 0 0 179 0 208m 2.85g 22m 1.2 0 0|0 0|0 252k 23k 2 17:13:17 0 0 974 0 0 975 0 208m 2.85g 23m 12.3 0 0|0 0|0 1m 121k 2 17:13:18 0 0 920 0 0 921 0 208m 2.85g 26m 10.1 0 0|0 0|0 1m 114k 2 17:13:19 0 0 810 0 0 811 0 208m 2.85g 26m 8.1 0 0|0 0|0 1m 100k 2 17:13:20 0 0 612 0 0 613 0 208m 2.85g 27m 8.1 0 0|0 0|0 798k 76k 2 17:13:21 0 0 0 0 0 1 0 208m 2.85g 27m 0 0 0|0 0|0 62b 1k 2 17:13:22 0 0 0 0 0 1 0 208m 2.85g 27m 0 0 0|0 0|0 62b 1k 2 17:13:23 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 30 0 0 31 0 208m 2.85g 27m 1.1 0 0|0 0|0 46k 5k 2 17:13:24 0 0 852 0 0 853 0 208m 2.85g 27m 14.8 0 0|0 0|0 1m 106k 2 17:13:25 0 0 768 0 0 769 0 208m 2.85g 29m 9.5 0 0|0 0|0 1m 95k 2 17:13:26 0 0 867 0 0 868 0 208m 2.85g 32m 13.4 0 0|0 0|0 1m 107k 2 17:13:27 0 0 705 0 0 706 0 208m 2.85g 31m 10.8 0 0|0 0|0 936k 88k 2 17:13:28 0 0 331 0 0 332 0 208m 2.85g 32m 3.9 0 0|0 0|0 446k 42k 2 17:13:29 0 0 0 0 0 1 0 208m 2.85g 32m 0 0 0|0 0|0 62b 1k 2 17:13:30 0 0 551 0 0 552 0 208m 2.85g 32m 6.1 0 0|0 0|0 739k 69k 2 17:13:31 0 0 728 0 0 729 0 208m 2.85g 35m 12.2 0 0|0 0|0 949k 90k 2 17:13:32 0 0 991 0 0 992 0 208m 2.85g 35m 11.7 0 0|0 0|0 1m 123k 2 17:13:33 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 935 0 0 935 0 208m 2.85g 38m 10.9 0 0|0 0|1 1m 116k 2 17:13:34 0 0 431 0 0 433 0 208m 2.85g 39m 5.9 0 0|0 0|0 563k 54k 2 17:13:35 0 0 0 0 0 1 0 208m 2.85g 39m 0 0 0|0 0|0 62b 1k 2 17:13:36 0 0 518 0 0 519 0 208m 2.85g 40m 5 0 0|0 0|0 693k 65k 2 17:13:37 0 0 904 0 0 905 0 208m 2.85g 40m 8.9 0 0|0 0|0 1m 112k 2 17:13:38 0 0 958 0 0 959 0 208m 2.85g 41m 14.2 0 0|0 0|0 1m 119k 2 17:13:39 0 0 899 0 0 900 0 208m 2.85g 44m 11.1 0 0|0 0|0 1m 111k 2 17:13:40 0 0 401 0 0 402 0 208m 2.85g 42m 5.9 0 0|0 0|0 540k 50k 2 17:13:41 0 0 856 0 0 857 0 208m 2.85g 44m 9.4 0 0|0 0|0 1m 106k 2 17:13:42 0 0 232 0 0 233 0 208m 2.85g 45m 2.9 0 0|0 0|0 311k 29k 1 17:13:43 Source DB: insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 2 0 12 1 13 0 56g 115g 2.82g 52.2 0 0|0 0|1 1k 822k 2 17:13:02 0 0 0 42 0 43 0 56g 115g 2.65g 88.1 0 0|0 0|1 5k 4k 2 17:13:03 0 0 0 14 0 15 1 56g 115g 1.88g 101 0 0|0 0|1 1k 2k 2 17:13:04 0 0 0 33 1 35 0 56g 115g 1.89g 53.8 16.6 0|0 1|0 4k 4k 2 17:13:05 0 0 0 0 0 1 0 56g 115g 1.9g 0 0 0|0 1|0 62b 1k 2 17:13:06 0 0 0 34 0 35 0 56g 115g 1.9g 47.3 0 0|0 0|0 4k 4m 2 17:13:07 0 0 0 91 0 92 0 56g 115g 1.89g 86.8 0 0|0 0|0 12k 8k 2 17:13:08 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 149 0 150 0 56g 115g 1.9g 82.5 0 0|0 0|0 20k 13k 2 17:13:09 0 0 0 287 0 287 0 56g 115g 1.9g 74.3 0 0|0 0|1 38k 25k 2 17:13:10 0 0 0 387 0 389 0 56g 115g 1.91g 64.6 0 0|0 0|0 52k 33k 2 17:13:11 0 0 0 580 0 581 0 56g 115g 1.93g 59 0 0|0 0|0 78k 49k 2 17:13:12 0 0 0 685 0 686 0 56g 115g 1.93g 47.9 0 0|0 0|0 93k 58k 2 17:13:13 0 0 0 729 0 730 0 56g 115g 1.95g 42.6 0 0|0 0|0 99k 61k 2 17:13:14 0 0 0 551 1 552 0 56g 115g 1.97g 34.5 0 0|0 1|0 74k 47k 2 17:13:15 0 0 0 0 0 1 0 56g 115g 1.98g 0 0 0|0 1|0 62b 1k 2 17:13:16 0 0 0 368 0 368 0 56g 115g 1.96g 20 0 0|0 0|1 50k 4m 2 17:13:17 0 0 0 1032 0 1034 0 56g 115g 2g 35.9 0 0|0 0|0 140k 87k 2 17:13:18 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 834 0 835 0 56g 115g 2.01g 42.3 0 0|0 0|0 113k 70k 2 17:13:19 0 0 0 922 0 922 0 56g 115g 2.02g 38.9 0 0|0 0|1 125k 77k 2 17:13:20 0 0 0 338 1 340 0 56g 115g 2.03g 13 0 0|0 1|0 46k 29k 2 17:13:21 0 0 0 0 0 1 0 56g 115g 2.04g 0 0 0|0 1|0 62b 1k 2 17:13:22 0 0 0 0 0 1 0 56g 115g 2.04g 0 0 0|0 1|0 62b 1k 2 17:13:23 0 0 0 185 0 186 0 56g 115g 2.02g 24.7 0 0|0 0|0 25k 4m 2 17:13:24 0 0 0 920 0 920 0 56g 115g 2.02g 29.9 0 0|0 0|1 125k 77k 2 17:13:25 0 0 0 741 0 742 0 56g 115g 2.04g 49 0 0|0 0|1 100k 62k 2 17:13:26 0 0 0 886 0 887 0 56g 115g 2.05g 33 0 0|0 0|1 120k 74k 2 17:13:27 0 0 0 683 0 684 0 56g 115g 2.08g 57.3 0 0|0 0|1 92k 58k 2 17:13:28 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 138 1 140 0 56g 115g 2.06g 14.5 0 0|0 1|0 18k 12k 2 17:13:29 0 0 0 12 0 12 0 56g 115g 2.07g 0.2 0 0|0 0|1 1k 4m 2 17:13:30 0 0 0 761 0 763 0 56g 115g 2.08g 46.5 0 0|0 0|0 103k 64k 2 17:13:31 0 0 0 762 0 762 0 56g 115g 2.09g 47 0 0|0 0|1 103k 64k 2 17:13:32 0 0 0 957 0 959 0 56g 115g 2.1g 35.3 0 0|0 0|0 130k 80k 2 17:13:33 0 0 0 905 0 905 0 56g 115g 2.13g 38.5 0 0|1 0|1 123k 76k 2 17:13:34 0 0 0 239 1 241 0 56g 115g 2.12g 14.9 0 0|0 1|0 32k 21k 2 17:13:35 0 0 0 0 0 1 0 56g 115g 2.13g 0 0 0|0 1|0 62b 1k 2 17:13:36 0 0 0 780 0 781 0 56g 115g 2.13g 42.8 0 0|0 0|0 106k 4m 2 17:13:37 0 0 0 805 0 806 0 56g 115g 2.14g 39.8 0 0|0 0|0 109k 68k 2 17:13:38 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 988 0 989 0 56g 115g 2.16g 34.8 0 0|0 0|0 134k 83k 2 17:13:39 0 0 0 953 0 953 0 56g 115g 2.17g 39.9 0 0|0 0|1 129k 80k 2 17:13:40 0 0 0 375 1 376 0 56g 115g 2.18g 13.9 0 0|1 0|1 51k 1m 2 17:13:41 0 0 0 867 0 869 0 56g 115g 2.19g 41.7 0 0|0 0|0 118k 73k 1 17:13:42 Total: MongoDB shell version: 2.0.1 connecting to: localhost:27017/prodcopy archiving clicks for ad_id: 4d6c12497b90420373000056 archiving documents... archived 19045 documents in 40701ms. For the queued approach with this same set up, I was able to queue ~1200 documents per second, and the archiving process performed at about the same ~500 documents per second. Example output from mongostat while this was running: Archive DB: insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:44 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:45 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:46 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:47 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:48 0 0 0 0 0 1 0 208m 2.85g 18m 0 0 0|0 0|0 62b 1k 2 17:16:49 0 0 248 0 0 249 0 208m 2.85g 17m 6 0 0|0 0|0 283k 31k 2 17:16:50 0 0 780 0 0 781 0 208m 2.85g 18m 8.2 0 0|0 0|0 883k 97k 2 17:16:51 0 0 1256 0 0 1257 0 208m 2.85g 22m 13.7 0 0|0 0|0 1m 155k 2 17:16:52 0 0 1084 0 0 1085 0 208m 2.85g 21m 10 0 0|0 0|0 1m 134k 2 17:16:53 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 1108 0 0 1109 0 208m 2.85g 25m 9.8 0 0|0 0|0 1m 137k 2 17:16:54 0 0 1108 0 0 1109 0 208m 2.85g 24m 9.8 0 0|0 0|0 1m 137k 2 17:16:55 0 0 1110 0 0 1111 0 208m 2.85g 27m 11 0 0|0 0|0 1m 137k 2 17:16:56 0 0 1166 0 0 1167 0 208m 2.85g 31m 11.3 0 0|0 0|0 1m 144k 2 17:16:57 0 0 1246 0 0 1247 0 208m 2.85g 31m 10.6 0 0|0 0|0 1m 154k 2 17:16:58 0 0 1094 0 0 1095 0 208m 2.85g 35m 9.6 0 0|0 0|0 1m 135k 2 17:16:59 0 0 957 0 0 958 0 208m 2.85g 33m 9.6 0 0|0 0|0 1m 119k 2 17:17:00 0 0 721 0 0 722 0 464m 3.35g 38m 8.9 0 0|0 0|0 1m 90k 2 17:17:01 0 0 243 0 0 243 0 464m 3.35g 34m 9.7 0 0|0 0|0 445k 31k 2 17:17:02 0 0 921 0 0 923 0 464m 3.35g 37m 30.1 0 0|0 0|0 1m 114k 2 17:17:03 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 668 0 0 669 0 464m 3.35g 38m 9.6 0 0|0 0|0 962k 83k 2 17:17:04 0 0 995 0 0 995 0 464m 3.35g 41m 21.2 0 0|0 0|1 1m 123k 2 17:17:05 0 0 467 0 0 468 0 464m 3.35g 27m 12 0 0|1 0|1 679k 58k 2 17:17:06 0 0 1 0 0 2 1 464m 3.35g 17m 85.5 0 0|1 0|1 1k 1k 2 17:17:07 0 0 337 0 0 338 0 464m 3.35g 19m 36.2 0 0|0 0|1 499k 42k 2 17:17:08 0 0 917 0 0 919 0 464m 3.35g 21m 20.4 0 0|0 0|0 1m 114k 2 17:17:09 0 0 1022 0 0 1022 0 464m 3.35g 25m 22.7 0 0|0 0|1 1m 126k 2 17:17:10 0 0 970 0 0 971 0 464m 3.35g 24m 27.4 0 0|0 0|1 1m 120k 2 17:17:11 0 0 166 0 0 168 0 464m 3.35g 25m 1.2 0 0|0 0|0 251k 21k 2 17:17:12 0 0 862 0 0 863 0 464m 3.35g 28m 23.1 0 0|0 0|0 1m 107k 2 17:17:13 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 888 0 0 889 0 464m 3.35g 28m 28.8 0 0|0 0|0 1m 110k 2 17:17:14 0 0 540 0 0 541 0 464m 3.35g 30m 12.2 0 0|0 0|0 802k 67k 2 17:17:15 0 0 879 0 0 880 0 464m 3.35g 31m 22.3 0 0|0 0|0 1m 109k 2 17:17:16 0 0 473 0 0 474 0 464m 3.35g 36m 17.2 0 0|0 0|0 1m 59k 1 17:17:17 Source DB: insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 0 0 1 0 56g 115g 1.86g 0 0 0|0 0|0 62b 1k 1 17:16:29 0 0 0 0 0 1 0 56g 115g 1.86g 0 0 0|0 0|0 62b 1k 1 17:16:30 1 1 1 0 1 3 0 56g 115g 1.87g 51.1 0 0|0 0|1 475b 697k 2 17:16:31 0 0 0 0 0 1 0 56g 115g 349m 129 0 0|0 0|1 62b 1k 2 17:16:32 0 0 0 0 0 1 0 56g 115g 367m 101 0 0|0 0|1 62b 1k 2 17:16:33 0 0 0 0 0 1 0 56g 115g 384m 73.4 0 0|0 0|1 62b 1k 2 17:16:34 0 0 0 0 0 1 0 56g 115g 428m 112 0 0|0 0|1 62b 1k 2 17:16:35 0 0 0 0 0 1 0 56g 115g 436m 94.8 0 0|0 0|1 62b 1k 2 17:16:36 0 0 0 0 0 1 0 56g 115g 450m 108 0 0|0 0|1 62b 1k 2 17:16:37 0 0 0 0 0 1 0 56g 115g 461m 110 0 0|0 0|1 62b 1k 2 17:16:38 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 0 0 1 0 56g 115g 480m 95.5 0 0|0 0|1 62b 1k 2 17:16:39 0 0 0 0 0 1 0 56g 115g 494m 73.5 0 0|0 0|1 62b 1k 2 17:16:40 0 0 0 0 0 1 0 56g 115g 531m 120 0 0|0 0|1 62b 1k 2 17:16:41 0 0 0 0 0 1 0 56g 115g 541m 108 0 0|0 0|1 62b 1k 2 17:16:42 0 0 0 0 0 1 0 56g 115g 564m 74.6 0 0|0 0|1 62b 1k 2 17:16:43 0 0 0 0 0 1 0 56g 115g 579m 127 0 0|0 0|1 62b 1k 2 17:16:44 0 0 0 0 0 1 0 56g 115g 605m 100 0 0|0 0|1 62b 1k 2 17:16:45 0 0 0 0 0 1 0 56g 115g 631m 98.5 0 0|0 0|1 62b 1k 2 17:16:46 0 0 0 0 0 1 0 56g 115g 657m 89.1 0 0|0 0|1 62b 1k 2 17:16:47 0 0 0 0 0 1 0 56g 115g 662m 87.8 0 0|0 0|1 62b 1k 2 17:16:48 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 0 0 1 0 56g 115g 680m 93.5 0 0|0 0|1 62b 1k 2 17:16:49 0 1 0 530 1 531 0 56g 115g 767m 81.3 0 0|0 0|1 72k 4m 2 17:16:50 0 0 0 884 0 886 0 56g 115g 721m 48 0 0|0 0|0 120k 74k 2 17:16:51 0 0 0 1079 0 1080 0 56g 115g 698m 33.6 0 0|0 0|0 146k 90k 2 17:16:52 0 0 0 1294 0 1294 0 56g 115g 720m 23.8 0 0|1 0|1 175k 108k 2 17:16:53 0 0 0 1118 1 1120 0 56g 115g 740m 31.9 0 0|0 0|0 152k 4m 2 17:16:54 0 0 0 1111 0 1112 0 56g 115g 718m 33.3 0 0|0 0|0 151k 93k 2 17:16:55 0 0 0 1097 0 1098 0 56g 115g 706m 33.1 0 0|0 0|0 149k 92k 2 17:16:56 0 0 0 1097 0 1098 0 56g 115g 701m 33.2 0 0|0 0|0 149k 92k 2 17:16:57 0 0 0 1091 1 1092 0 56g 115g 696m 35.1 0 0|0 0|0 148k 4m 2 17:16:58 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 1149 0 1150 0 56g 115g 718m 21.3 0 0|0 0|0 156k 96k 2 17:16:59 0 0 0 1091 0 1091 0 56g 115g 707m 31.2 0 0|0 0|1 148k 91k 2 17:17:00 0 0 0 468 0 470 0 56g 115g 714m 9.3 0 0|0 0|0 63k 40k 2 17:17:01 0 0 0 332 0 333 0 56g 115g 694m 27.3 0 0|0 0|0 45k 28k 2 17:17:02 0 0 0 1013 1 1014 0 56g 115g 713m 17.8 0 0|0 0|0 137k 4m 2 17:17:03 0 0 0 551 0 552 0 56g 115g 691m 39.9 0 0|0 0|0 74k 47k 2 17:17:04 0 0 0 1140 0 1141 0 56g 115g 709m 19.7 0 0|0 0|0 155k 95k 2 17:17:05 0 0 0 126 0 127 0 56g 115g 709m 2.1 0 0|0 0|0 17k 11k 2 17:17:06 0 0 0 92 0 92 0 56g 115g 710m 1.6 0 0|0 0|1 12k 8k 2 17:17:07 0 0 0 605 1 607 0 56g 115g 700m 29.1 0 0|0 0|0 82k 4m 2 17:17:08 insert query update delete getmore command flushes mapped vsize res locked % idx miss % qr|qw ar|aw netIn netOut conn time 0 0 0 814 0 814 0 56g 115g 699m 17.3 0 0|0 0|1 110k 68k 2 17:17:09 0 0 0 1036 0 1037 0 56g 115g 708m 18.4 0 0|0 0|0 140k 87k 2 17:17:10 0 0 0 820 0 821 0 56g 115g 715m 14.4 0 0|1 0|1 111k 69k 2 17:17:11 0 0 0 298 0 300 0 56g 115g 683m 24.2 0 0|0 0|0 40k 26k 2 17:17:12 0 0 0 907 1 908 0 56g 115g 699m 20.4 0 0|0 0|0 123k 4m 2 17:17:13 0 0 0 946 0 947 0 56g 115g 707m 16.4 0 0|0 0|0 128k 79k 2 17:17:14 0 0 0 374 0 375 0 56g 115g 677m 36.7 0 0|0 0|0 50k 32k 2 17:17:15 0 0 0 905 1 905 0 56g 115g 689m 16.2 0 0|0 0|0 123k 821k 2 17:17:16 0 0 0 259 0 261 0 56g 115g 690m 5.1 0 0|0 0|0 35k 22k 1 17:17:17 0 0 0 0 0 1 0 56g 115g 690m 0 0 0|0 0|0 62b 1k 1 17:17:18 Total: MongoDB shell version: 2.0.1 connecting to: localhost:27017/prodcopy archiving clicks for ad_id: 4d6c12497b90420373000062 ensuring sparse index on field "arch" marking documents for archive... queued 22227 documents for archive in 19369ms. archiving queued documents to archive.clicks... archived 22227 queued documents to archive.clicks in 26901ms. Upgrading the hard drive to an SSD, I was able to almost triple the performance of the archiving step, and archive ~1300 documents per second. Obviously this is all a giant hack, and it would be nice if MongoDB supported some kind of partitioning feature so we didn’t have to do any of this manually. But until then, this is the best I could get given the tools currently available. Source: http://blog.brianploetz.com/post/12131083486/adventures-in-archiving-with-mongodb

Lynden is a family of transportation and logistics companies specialized in shipping to Alaska and other locations worldwide. Over land, on the water, in the air - or in any combination - Lynden has been helping customers solve transportation problems for over a century. The Lynden Freight Management System is a NetBeans Platform application which serves a dual purpose as both a planning and freight tracking tool. The Planning module allows terminal managers to see all freight that is currently inbound to their location as well as freight that is scheduled to depart from their location so they can make the most efficient use of their dock space and resources as possible. The Trace module allows customer service personnel to search for customer account information, view the tracking history of any given freight item in the system as well as display any documents related to the shipment, such as bills of lading or delivery receipts. NetBeans Platform Lynden has benefited from the NetBeans Platform as it allows developers to focus on the business logic of our applications rather than the underlying "plumbing". We are able to leverage built-in support for event handling, enable/disable functionality on UI controls, dockable windows, and automatic updates for our application with minimal work compared to rolling our own framework. We chose to go the desktop application route as we have a number of existing desktop applications here that this application will likely need to interface with at some point, as well as a commercial set of rich UI components that we have been using for some time now. For the initial deployment, we will be pushing the installer out to employee PCs via the Landesk remote desktop administration tool. Future updates to various modules within the application will be done via the update center functionality built into the NetBeans Platform. Screenshots

mock is a library for testing in python. it allows you to replace parts of your system under test with mock objects. the main feature of mock is that it's simple to use, but mock also makes possible more complex mocking scenarios. this is my philosophy of api design as it happens: simple things should be simple but complex things should be possible . several of these more complex scenarios are shown in the further examples section of the documentation. i've just updated one of these, the example of mocking properties. properties in python are descriptors . when they are fetched from the class of an object they trigger code that is then executed. the code that is executed is the method that you have wrapped as the property getter. note that there is a special rule for attribute lookup on certain types of descriptors, which include properties. even if an instance attribute exists, the class attribute will still be used instead. this is an exception to the normal attribute lookup rule that instance attributes are fetched in preference to class attributes. this is important because it means that when you want to mock a property you have to do it on the class and can't simply stick an attribute onto an object. if you're using mock 0.7, with its support for magic methods , we can patch the property name and add a __get__ method to our mock. the presence of the __get__ method makes it a descriptor, so we can use it as a property: >>> from mock import mock, patch >>> class foo(object): ... @property ... def fish(self): ... return 'fish' ... >>> with patch.object(foo, 'fish') as mock_fish: ... mock_fish.__get__ = mock(return_value='mocked fish') ... foo = foo() ... print foo.fish ... mocked fish >>> mock_fish.__get__.assert_called_with(mock_fish, foo, foo) in this example mock_fish replaces the property and the mock we put in place of __get__ becomes the mocked getter method. as we're patching on the class this affects all instances of foo . there's no point in using magicmock for this. magicmock normallly makes using the python protocol methods simpler by preconfiguring them. as you can see from the example above, mocking __get__ is supported but it isn't hooked up by default. it wouldn't be helpful if mocking any method on a class replaced it with a mock that acted as a descriptor, so if you want a mock to behave as a descriptor then you have to configure __get__ , __set__ and __delete__ yourself. here's an alternative approach that works with all recent-ish versions of mock: >>> from mock import mock, patch >>> class propertymock(mock): ... def __get__(self, instance, owner): ... return self() ... >>> prop_mock = propertymock() >>> with patch.object(foo, 'fish', prop_mock): ... foo = foo() ... prop_mock.return_value = 'mocked fish' ... print foo.fish ... mocked fish >>> prop_mock.assert_called_with() as an added bonus, both of these examples work even if the foo instance is created outside of the patch block. so long as the code using the property is executed whilst the patch is in place the attribute lookup will find our mocked version. source: http://www.voidspace.org.uk/python/weblog/arch_d7_2011_06_04.shtml

Let me first define what we mean by circular dependency in OOAD terms vis-a-vis Java. Suppose we have a class called A which has class B’s Object. (in UML terms A HAS B). at the same time we class B is also composed of Object of class A (in UML terms B HAS A). obviously this represents circular dependency because while creating the object of A, the compiler must know the size of B... on the other hand while creating object of B, the compiler must know the size of A. this is something like egg vs. chicken problem... this may be possible in real life situation as well. for example suppose a multi storied building has a lift. so in the UML terms, the building HAS lift... but at the same time, suppose, while constructing the lift object, we need to give it the information about the building object to access various functionalities of the Building class... for example, suppose the speed of the lift is set depending on the number of floors of the Building... hence while constructing the Lift object it must access the functionalities of the Building object which will give the number of floors the building has got...hence in UML terms the lift HAS building... so this is a sure case of circular dependency... in real java code it will look something as follows: public class Building { private Lift lift; private int floor; public Building(){ lift = new Lift(); setFloor(15); } public int getFloor(){ return floor; } public void setFloor(int floor){ this.floor = floor; } }//end of class building //class Lift public class Lift { private Building building; private int Speed; public Lift(){ building = new Building(); setSpeed(); } public void setSpeed(){ if (building.getFloor()>20){ //one set of functionalities //may be the the speed of the lift will be more this.Speed = 10; } else { //different set of functionalities //may be the speed of the lift will be less this.Speed = 5; } } public int getSpeed(){ return Speed; } }//end of class Lift As it becomes clear from the above code, that while creating the Building object it will create the Lift object, and while creating the Lift object, it will try to create a Building object to access some of its functionalities... So, ultimately it will go out of memory and we get a StackOverflow runtime exception... So how do we handle this problem in Java? We actually tackle this problem by declaring an IBuildingProxy interface and by deriving our Building class from that... the lift class, instead of Having Building object, it Has IBuildingProxy... the source code of the solution looks like the following... public interface IBuildingProxy { int getFloor(); void setFloor(int floor); } public class Building implements IBuildingProxy{ private Lift lift; private int floor; public Building(){ lift = new Lift(this); setFloor(15); } public int getFloor(){ return floor; } public void setFloor(int floor){ this.floor = floor; } } public class Lift { private IBuildingProxy building; private int Speed; public Lift(Building b){ this.building = b; setSpeed(); } private void setSpeed(){ if (building.getFloor()>20){ / /one set of functionalities //may be the the speed of the lift will be more this.Speed = 10; } else { //different set of functionalities //may be the speed of the lift will be less this.Speed = 5; } } public int getSpeed(){ return Speed; } } public class CircularDependencyTest { public static void main(String[] args){ Building b = new Building(); Lift l = new Lift(b); } } So whats the principle behind such work around... It will be clear soon... As it becomes clear from the code that Building HAS Lift... That is not a problem... Now when it comes to solve the part that Lift HAS Building, instead of the Building object, we have created an IBuildingProxy interface and we pass it to the Lift class... what it essentially means, that the building class knows the memory requirement to initialize the Lift object, and as the Lift class HAS just a proxy interface of the Building, it does not have to care for the Building's memory requirement... and that solves the problem... Hope this discussion becomes helpful for Java learners...