Today I am going to discuss about creating a Grid in Android having image and associated checkbox. The idea is to have Grid with clickable images and checkbox. When an image is clicked the corresponding checkbox state is toggled. It is relatively easy to add custom items to the Grid. But its bit tricky to add clickable images and check/uncheck checkbox on click of the image. If you want to make the images clickable and check/uncheck the checkbox on image click, that is not that straightforward. In this article, I will take you through the creation of the Grid with clickable images and checkboxes. Creating a new project Open Eclipse and create a new android project. Choose a blank activity for the project. Create main Grid layout Open the layout file for the main activity and add a GridView component to it. Create Custom layout for grid items The next step is to create a custom layout that will be used to represent each item of the grid. Here we will define a custom layout using an ImageView and CheckBox android components. Create a new layout file named griditem.xml and add ImageView and CheckBox components to it as shown below: Adding Images to project For this sample project I am adding images to the project itself for use in the grid. But you can source those from database or any other content provider supported by android. In Android project there are few drawable-* folders under res folder. These are used for storing various resources used by the project. We can put images in any of these folders and Android SDK will copy them to other folders. If you have different images for different resolutions you can copy them to respective folders and correct image would be picked up based on the device resolution. Another option is to create a drawable folder under res and add images there. These will be copied to all drawable-* folders. We will use this approach here. Create custom adapter The next important step is to create a custom adapter which uses custom layout that we defined earlier to add items to the grid. The custom adapter should extend the BaseAdapter. The getView()method of the adapter is invoked for each item of the grid. In this method we assign values to the ImageView and checkbox. The listeners on the custom grid items are added here only. The code for getView() looks like the following: @Override public View getView(int position, View convertView, ViewGroup parent) { ViewHolder holder; if (convertView == null) { holder = new ViewHolder(); convertView = mInflater.inflate( R.layout.griditems, null); holder.imageview = (ImageView) convertView.findViewById(R.id.grid_item_image); holder.checkbox = (CheckBox) convertView.findViewById(R.id.grid_item_checkbox); convertView.setTag(holder); } else { holder = (ViewHolder) convertView.getTag(); } holder.checkbox.setId(position); holder.imageview.setId(position); holder.checkbox.setOnClickListener(new OnClickListener() { public void onClick(View v) { CheckBox cb = (CheckBox) v; int id = cb.getId(); if (thumbnailsselection[id]){ //cb.setChecked(false); thumbnailsselection[id] = false; } else { //cb.setChecked(true); thumbnailsselection[id] = true; } } }); holder.imageview.setOnClickListener(new ImageClickListener(context, holder,thumbnailsselection)); holder.imageview.setImageResource(thumbnails[position]); holder.checkbox.setChecked(thumbnailsselection[position]); holder.id = position; return convertView; } Here we are saving the state of checkboxes in an array, thumbnailsselection. Since checkboxes are being added dynamically and there is no unique identifier for them, this will help in identifying the current state of checkbox whose state has to be toggled when an image is clicked. For the same reason we need to create a separate ImageClickListener. As we cannot identify the checkbox on click of corresponding image, we have to create the separate listener which holds the reference to holder class. Using this holder instance we can check/uncheck the checkbox when a corresponding image is clicked. Sometimes when you click on the image, the ImageClickListener is not invoked. The reason for this is that the checkbox overrides the focus event of container grid. Hence checkbox takes the focus when any item of the grid is clicked. To solve this issue you need to add the following properties to the checkbox. android:focusable=“false” android:focusableInTouchMode=“false” References Image icons source: http://www.iconarchive.com http://developer.android.com/

while searching on the internet on how to make sip video calls using c#, i recognised that there aren’t any brief and straightforward tutorial in this topic. i found multi-page articles (sorry, but some of them are full of bullsh*t) and neverending forum threads, but none of them provided me complete solution. therefore, i undertook to create a short and concise guide on how to make video calls in c# using the voip technology. look at the prerequisites: pbx: to be able to make and receive video calls you, a phone system (e.g. asterisk) is essentially needed. you need to add a new sip account in your pbx for this application. visual studio: this solution is based on a console softphone, so a new visual c# console application is just enough. voipsdk.dll: i used prewritten voip components to implement the sip video calling feature. the necessary .dll file can be found on this website: c# voip stack . it should be added to your references. test phone: to test your application you can use any voip phone stat supports video calling (e.g. bria softphone). now take a look at the code. as you can see below, just a few lines of c# code are enough to connect the application to a pbx and to initiate a video call. firstly, you need to perform the sip registration tasks. you need to create a softphone and a phone line object, then you need to specify the sip account to be used for the phone line. these configurations are needed to be able to register to your pbx. after calling the registerphoneline method the regsitration procedure starts, and the application will indicates its status due to the mysoftphone_phonelinestatechanged method. the phonecallvideosender and phonecallvideoreceiver objects are responsible for video handling. to handle the usb webcamera, the webcamera object can be used. the calltype class is used to identify whether the call is a video or an audio call. test the program: to make a test call, provide valid sip account details for this console application to be able to register to your pbx, then specify a telephone number to be dialled (it can be an other sip account that has been previously registered to the pbx). after running the application, it dials the provided phone number automatically meanwhile sending the image of the webcamera. using system; using ozeki.media.mediahandlers; using ozeki.media.mediahandlers.video; using ozeki.voip; using ozeki.voip.sdk; namespace video_call { internal class program { private static isoftphone softphone; // softphone object private static iphoneline phoneline; // phoneline object private static iphonecall call; private static string numbertodial; private static mediaconnector mediaconnector; private static void main(string[] args) { // create a softphone object with rtp port range 5000-10000 softphone = softphonefactory.createsoftphone(5000, 10000); // sip account registration data, (supplied by your voip service provider) var registrationrequired = true; var username = "444"; var displayname = "444"; var authenticationid = "444"; var registerpassword = "444"; var domainhost = "192.168.115.25"; var domainport = 5060; var account = new sipaccount(registrationrequired, displayname, username, authenticationid, registerpassword, domainhost, domainport); // send sip regitration request registeraccount(account); // prevents the termination of the application console.readline(); } static void registeraccount(sipaccount account) { try { phoneline = softphone.createphoneline(account); phoneline.registrationstatechanged += sipaccount_regstatechanged; softphone.registerphoneline(phoneline); } catch (exception ex) { console.writeline("error during sip registration: " + ex); } } private static void sipaccount_regstatechanged(object sender, registrationstatechangedargs e) { console.writeline(e.state); if (e.state == regstate.registrationsucceeded) createcall(); } static void createcall() { //numbertodial = "333"; //call = softphone.createcallobject(phoneline, numbertodial); var dialparams = new dialparameters("333"); dialparams.calltype = calltype.audiovideo; call = softphone.createcallobject(phoneline, dialparams); mediaconnector = new mediaconnector(); var phonecallvideosender = new phonecallvideosender(); var cam = webcamera.getdefaultdevice(); if (cam != null) { cam.start(); mediaconnector.connect(cam, phonecallvideosender); } phonecallvideosender.attachtocall(call); call.callstatechanged += call_callstatechanged; call.start(); } static void call_callstatechanged(object sender, callstatechangedargs e) { console.writeline("\ncall state: {0}.", e.state); } } }

Recently I had a dispute with my colleagues regarding performance penalty of lazy vals in Scala. It resulted in a set of microbenchmarks which compare lazy and non-lazy vals performance. All the sources can be found at http://git.io/g3WMzA. But before going to the benchmark results let's try to understand what can cause the performance penalty. For my JMH benchmark I created a very simple Scala class with lazy val in it: @State(Scope.Benchmark) class LazyValCounterProvider { lazy val counter = SlowInitializer.createCounter() } Now let's take a look at what is hidden under the hood of lazy keyword. At first, we need to compile given code with scalac, and then it can be decompiled to correspondent Java code. For this sake I used JD decompiler. It produced the following code: @State(Scope.Benchmark) @ScalaSignature(bytes="...") public class LazyValCounterProvider { private SlowInitializer.Counter counter; private volatile boolean bitmap$0; private SlowInitializer.Counter counter$lzycompute() { synchronized (this) { if (!this.bitmap$0) { this.counter = SlowInitializer.createCounter(); this.bitmap$0 = true; } return this.counter; } } public SlowInitializer.Counter counter() { return this.bitmap$0 ? this.counter : counter$lzycompute(); } } As it's seen, the lazy keyword is translated to a classical double-checked locking idiom for delayed initialization. Thus, most of the time the only performance penalty may come from a single volatile read per lazy val read (except for the time it takes to initialize lazy val instance since its very first usage). Let's finally measure its impact in numbers. My JMH-based microbenchmark is as simple as: public class LazyValsBenchmarks { @Benchmark public long baseline(ValCounterProvider eagerProvider) { return eagerProvider.counter().incrementAndGet(); } @Benchmark public long lazyValCounter(LazyValCounterProvider provider) { return provider.counter().incrementAndGet(); } } A baseline method access a final counter object and increments an integer value by 1 by calling incrementAndGet . And as we've just found out, the main benchmark method - lazyValCounter - in addition to what baseline method does also does one volatile read. Note: all measurements are performed on MBA with Core i5 1.7GHz CPU. All results were obtained by running JMH in a throughput mode. Both score and score error columns show operations/second. Each JMH run made 10 iterations and took 50 seconds. I performed 6 measurements with the different JVM and JMH options: client VM, 1 thread Benchmark Score Score error baseline 412277751.619 8116731.382 lazyValCounter 352209296.485 6695318.185 client VM, 2 threads Benchmark Score Score error baseline 542605885.932 15340285.497 lazyValCounter 383013643.710 53639006.105 client VM, 4 threads Benchmark Score Score error baseline 551105008.767 5085834.663 lazyValCounter 394175424.898 3890422.327 server VM, 1 thread Benchmark Score Score error baseline 407010942.139 9004641.910 lazyValCounter 341478430.115 18183144.277 server VM, 2 threads Benchmark Score Score error baseline 531472448.578 22779859.685 lazyValCounter 428898429.124 24720626.198 server VM, 4 threads Benchmark Score Score error baseline 549568334.970 12690164.639 lazyValCounter 374460712.017 17742852.788 The numbers show that lazy vals performance penalty is quite small and can be ignored in practice. For further reading about the subject I would recommend SIP 20 - Improved Lazy Vals Initialization, which contains very interesting in-depth analysis of existing issues with lazy initialization implementation in Scala.

This article follows on from my Omnipresent, Infallible, Omnipotent and Instantaneous Build Technologies one a couple of day ago. Specifically the last section: “Minimalist test execution, via hacks”, that addressed test times being very lengthy. I’ve made a proof of concept of that for Maven. The tests impacted by a change (a pending change specifically) can now be quickly determined and fed into the test runner to massively reduce test times. Massively reduced, this is, if you’ve managed to engineer hours of Selenium tests. Proof of concept I forked a Github project that Petri Kainulainen had made to discuss coverage for a Maven projects. Clone my fork, but cd into the “code-coverage-jacoco” directory. Running testimpact.sh there – it will rebuild a map of what tests cover what production-classes (sources). I’ve checked in the previous results from running this script, that’s what a team would do. It runs Maven against one test at a time, to calculate coverage then store that per test. Actually for Petri’s example, I’m only focussing on the integration tests (run by the “failsafe” plugin) rather than the unit tests (run by the “surefire” plugin). Even though Petri’s example project is not launching Selenium (or equivalent slow test), the integration test phase is where you would run that for a canonical Maven project. The testimpact.sh script uses python and ack (you’ll have to install those if you want to run this). I tried to use sqlite3’s CSV ingesting, but it was impossibly opaque, even with using StackOverflow’s best Questions/Answers, so I flipped to Python. Petri’s example uses JaCoCo for coverage, which spits out a handy CSV report (as well as HTML). There are some text files in src/integration-test/meta/ccexample that show the sources covered by each test. Yes, they are checked in. Those files end in .java but are actually plain text (sorry): There’s another file src/integration-test/impact-map.txt that contains a list of production sources and the tests that would exercise them. Actually its a map of sources vs tests: Experimenting with what I’ve done Change one of the two production classes in src/main/java/ccexample. Yes they are clones of each other – that’s just something I did after forking to increase the class count at the start. Don’t commit that pending change, just leave it there showing up as modified in git status. Run python tests.py and watch it run one or two tests in the same invocation. Now undo that change, and change the other source file, and run python tests.py again. Different tests ran, right? Undo that change, and do python tests.py once more. No tests ran, right? That would be the same for changed sources/classes that had no tests exercising them (covering them under test invocation) at all. Of course these few tests are really quick, but they could have been three subset from hundreds or thousands of tests with an elapsed time of many hours. Turning the idea into a solution It is also worth noting that scripts as I have them are not robust or optimized. There’s more source-control systems than just Git, of course. The script needs to be able to work for a commit too, not just a pending commit. The storage format for impact-map.txt will not scale, and you might want to excise certain categories of POJO if every test exercises them. To be correct in Maven-land, this should be a bunch of plugins that fit the Maven style. One plugin would be invoked from Jenkins and would probably run constantly if corresponding test times are up in the hours. That Jenkins job should check in changes to the meta-files as it sees changes. This is benign, and useful to team members who may want the shorter build. Would you check the impact map into source-control? It seems to me that the map data is related to the other source files. Perhaps if you took a historical view to things, the impact maps change with the source code at the same time. If you were bug-fixing something from the past (checking out a prior revision), you might be happy that the impact map also goes back in time. Of course there’s nothing here that couldn’t be stored in a key/value store, including the changing map over time. Or a service that could answer the question “what tests should I run if these source files are changed?” Except perhaps that uncommitted work (on your own workstation) isn’t going to be present in that store until after a commit, and the impact map data us updated to that into account. Taking the idea even further I’ve suggested that Selenium is the technology that would greatly lengthen test times. There are many other test technologies that are one, two or three orders slower than perfect unit tests. This idea is applicable to much more than just Selenium – the only requirement is to be able to measure coverage while individual tests are running. What I have done could also be extended to the “surefire” phase – almost identical to what I’ve done already – an opportunity for reuse of course. CI daemons like Jenkins could benefit from the same impact-driven test time reduction. At least the per-commit jobs that we do for the Continuous Delivery era of enterprise development. This idea could be extended to the test-method level. That would be harder still, but achievable in the same way. It comes with arguably negligible gains, though. You’d code it all, and work out that 5% wasn’t worth it (versus other ways of speeding up tests). We’d need Intellij and Eclipse plugins for this. The script needs to be able to work for range of commits as many teams batch them in in Jenkins-land. I wonder what is out there that already does this sort of thing already. Followups (Jan 13, 17, 2015) Markus Kohler notes: This is a great idea and I guess similar to what Google does (there was blog post about it, can’t find it ATM). But I think as it is it would not be completely correct. The reason is that as far as I can see, you just run the tests that use the modified class files. That is not enough, because in Java changing one source code file might result in other files needed to be recompiled. Gradle’s Java plugin (http://www.gradle.org/docs/current/userguide/java_plugin.html) computes that set of files for example for incremental compilation. As an example if a final static String is changed and that String is used in other classes, these classes has to be recompiled because the Java compiler inlines these kind of strings. Any Test that uses these classes would have to be re-run. He’s quite right there’s room for some edge case mistakes. To overcome those, you would need to check for changes that would lead to final static fields being inlined into other classes. Maybe a list of source files that, if changed, would cause all tests to be re-run (after a suitable warning).

After we introduced locked thread detection to Plumbr couple of months ago, we have started to receive queries similar to “hey, great, now I understand what is causing my performance issues, but what I am supposed to do now?” We are working hard to build the solution instructions into our own product, but in this post I am going to share several common techniques you can apply independent of the tool used for detecting the lock. The methods include lock splitting, concurrent data structures, protecting the data instead of the code and lock scope reduction. Locking is not evil, lock contention is Whenever you face a performance problem with the threaded code there is a chance that you will start blaming locks. After all, common “knowledge” is that locks are slow and limit scalability. So if you are equipped with this “knowledge” and start to optimize the code and getting rid of locks there is a chance that you end up introducing nasty concurrency bugs that will surface later on. So it is important to understand the difference between contended and uncontended locks. Lock contention occurs when a thread is trying to enter the synchronized block/method currently executed by another thread. This second thread is now forced to wait until the first thread has completed executing the synchronized block and releases the monitor. When only one thread at a time is trying to execute the synchronized code, the lock stays uncontended. As a matter of fact, synchronization in JVM is optimized for the uncontended case and for the vast majority of the applications, uncontended locks pose next to no overhead during execution. So, it is not locks you should blame for performance, but contended locks. Equipped with this knowledge, lets see what we can do to reduce either the likelihood of contention or the length of the contention. Protect the data not the code A quick way to achieve thread-safety is to lock access to the whole method. For example, take look at the following example, illustrating a naive attempt to build an online poker server: class GameServer { public Map<> tables = new HashMap>(); public synchronized void join(Player player, Table table) { if (player.getAccountBalance() > table.getLimit()) { List tablePlayers = tables.get(table.getId()); if (tablePlayers.size() < 9) { tablePlayers.add(player); } } } public synchronized void leave(Player player, Table table) {/*body skipped for brevity*/} public synchronized void createTable() {/*body skipped for brevity*/} public synchronized void destroyTable(Table table) {/*body skipped for brevity*/} } The intentions of the author have been good - when new players join() the table, there must be a guarantee that the number of players seated at the table would not exceed the table capacity of nine. But whenever such a solution would actually be responsible for seating players to tables - even on a poker site with moderate traffic, the system would be doomed to constantly trigger contention events by threads waiting for the lock to be released. Locked block contains account balance and table limit checks which potentially can involve expensive operations both increasing the likelihood and length of the contention. First step towards solution would be making sure we are protecting the data, not the code by moving the synchronization from the method declaration to the method body. In the minimalistic example above, it might not change much at the first place. But lets consider the whole GameServerinterface, not just the single join() method: class GameServer { public Map> tables = new HashMap>(); public void join(Player player, Table table) { synchronized (tables) { if (player.getAccountBalance() > table.getLimit()) { List tablePlayers = tables.get(table.getId()); if (tablePlayers.size() < 9) { tablePlayers.add(player); } } } } public void leave(Player player, Table table) {/* body skipped for brevity */} public void createTable() {/* body skipped for brevity */} public void destroyTable(Table table) {/* body skipped for brevity */} } What originally seemed to be a minor change, now affects the behaviour of the whole class. Whenever players were joining tables, the previously synchronized methods locked on theGameServer instance (this) and introduced contention events to players trying to simultaneouslyleave() tables. Moving the lock from the method signature to the method body postpones the locking and reduces the contention likelihood. Reduce the lock scope Now, after making sure it is the data we actually protect, not the code, we should make sure our solution is locking only what is necessary - for example when the code above is rewritten as follows: public class GameServer { public Map> tables = new HashMap>(); public void join(Player player, Table table) { if (player.getAccountBalance() > table.getLimit()) { synchronized (tables) { List tablePlayers = tables.get(table.getId()); if (tablePlayers.size() < 9) { tablePlayers.add(player); } } } } //other methods skipped for brevity } then the potentially time-consuming operation of checking player account balance (which potentially can involve IO operations) is now outside the lock scope. Notice that the lock was introduced only to protect against exceeding the table capacity and the account balance check is not anyhow part of this protective measure. Split your locks When we look at the last code example, you can clearly notice that the whole data structure is protected by the same lock. Considering that we might hold thousands of poker tables in this structure, it still poses a high risk for contention events as we have to protect each table separately from overflowing in capacity. For this there is an easy way to introduce individual locks per table, such as in the following example: public class GameServer { public Map> tables = new HashMap>(); public void join(Player player, Table table) { if (player.getAccountBalance() > table.getLimit()) { List tablePlayers = tables.get(table.getId()); synchronized (tablePlayers) { if (tablePlayers.size() < 9) { tablePlayers.add(player); } } } } //other methods skipped for brevity } Now, if we synchronize the access only to the same table instead of all the tables, we have significantly reduced the likelihood of locks becoming contended. Having for example 100 tables in our data structure, the likelihood of the contention is now 100x smaller than before. Use concurrent data structures Another improvement is to drop the traditional single-threaded data structures and use data structures designed explicitly for concurrent usage. For example, when picking ConcurrentHashMapto store all your poker tables would result in code similar to following: public class GameServer { public Map> tables = new ConcurrentHashMap>(); public synchronized void join(Player player, Table table) {/*Method body skipped for brevity*/} public synchronized void leave(Player player, Table table) {/*Method body skipped for brevity*/} public synchronized void createTable() { Table table = new Table(); tables.put(table.getId(), table); } public synchronized void destroyTable(Table table) { tables.remove(table.getId()); } } The synchronization in join() and leave() methods is still behaving as in our previous example, as we need to protect the integrity of individual tables. So no help from ConcurrentHashMap in this regards. But as we are also creating new tables and destroying tables in createTable() and destroyTable()methods, all these operations to the ConcurrentHashMap are fully concurrent, permitting to increase or reduce the number of tables in parallel. Other tips and tricks Reduce the visibility of the lock. In the example above, the locks are declared public and are thus visible to the world, so there is there is a chance that someone else will ruin your work by also locking on your carefully picked monitors. Check out java.util.concurrent.locks to see whether any of the locking strategies implemented there will improve the solution. Use atomic operations. The simple counter increase we are actually conducting in example above does not actually require a lock. Replacing the Integer in count tracking withAtomicInteger would most suit this example just fine. Hope the article helped you to solve the lock contention issues, independent of whether you are using Plumbr automatic lock detection solution or manually extracting the information from thread dumps.

Part 1 – Configuring Ehcache with Spring Ehcache is most widely used open source cache for boosting performance, offloading your database, and simplifying scalability. It is very easy to configure Ehcache with Spring context xml. Assuming you have a working Spring configured application, here we discuss how to integrate Ehcache with Spring. In order to configure it, we need to add a CacheManager into the Spring context file. Code snippet added below In ‘ehcache’ bean, we are referring to ‘Ehcache.xml’ file which will contain all ehcache configurations. Sample ehcache file is added below. You can configure this file according to your project needs. Here you can two caches, defaultCache and one defined with name ‘TestCache’. DefaultCache configuration is applied to any cache that is not explicitly configured. Part 2 - Enabling Caching Annotations In order to used Spring provided java annotations , we need to declarative enable cache annotations in Spring config file. Code snippet added below. In order to use above annotation declaration, we need to add cache namespace into the spring file. Code snippet added below. Spring Annotations for caching @Cacheable triggers cache population @CacheEvict triggers cache eviction @CachePut updates the cache without interfering with the method execution I will give brief description and practical implementation of these annotations. For detailed explanations please refer spring documentation for caching . @Cacheable - This annotation means that the return value of the corresponding method can be cached and multiple invocation of this method with same arguments must populate the return value from cache without executing the method. Code snippet added below. @Cacheable("customer") public Customer findCustomer(String custId) {...} Each time the method is called, the cache with name ‘customer’ is checked to see whether the invocation has been already executed for the same ‘custId’. @CachePut - This annotation is used to update the cache without interfering the method execution. Method will always be executed and its result will be replaced in the cache. This annoatation is used for cache population. @CachePut("customer") public Customer updateBook(String custId){...} @CacheEvict – This annotation is used for cache eviction, used to remove stale or unused data from the cache. @CacheEvict(value="customer", allEntries=true) public void unloadCustomer(String newBatch) With above configurations, all entries in the ‘customer’ cache will be removed. Above description will give head start to configure Ehcachewith spring and use some of the caching annotations.

Originally Written byAndrew Moore MySQL table alterations can interrupt production traffic causing bad customer experience or in worst cases, loss of revenue. Not all DBAs, developers, syadmins know MySQL well enough to avoid this pitfall. DBAs usually encounter these kinds of production interruptions when working with upgrade scripts that touch both application and database or if an inexperienced admin/dev engineer perform the schema change without knowing how MySQL operates internally. Truths * Direct MySQL ALTER table locks for duration of change (pre-5.6) * Online DDL in MySQL 5.6 is not always online and may incurr locks * Even with Percona Toolkit‘s pt-online-schema-change there are several workloads that can experience blocking Here on the Percona MySQL Managed Services team we encourage our clients to work with us when planning and performing schema migrations. We aim to ensure that we are using the best method available in their given circumstance. Our intentions to avoid blocking when performing DDL on large tables ensures that business can continue as usual whilst we strive to improve response time or add application functionality. The bottom line is that a business relying on access to its data cannot afford to be down during core trading hours. Many of the installations we manage are still below MySQL 5.6, which requires us to seek workarounds to minimize the amount of disruption a migration can cause. This may entail slave promotion or changing the schema with an ‘online schema change’ tool. MySQL version 5.6 looks to address this issue by reducing the number of scenarios where a table is rebuilt and locked but it doesn’t yet cover all eventualities, for example when changing the data type of a column a full table rebuild is necessary. The topic of 5.6 Online Schema Change was discussed in great detail last year in the post, “Schema changes – what’s new in MySQL 5.6?” by Przemysław Malkowski With new functionality arriving in MySQL 5.7, we look forward to non-blocking DDL operations such as; OPTIMIZE TABLE and RENAME INDEX. (More info) The best advice for MySQL 5.6 users is to review the matrix to familiarize with situations where it might be best to look outside of MySQL to perform schema changes, the good news is that we’re on the right path to solving this natively. Truth be told, a blocking alter is usually going to go unnoticed on a 30MB table and we tend to use a direct alter in this situation, but on a 30GB or 300GB table we have some planning to do. If there is a period of time where activity is low and the this is permissive of locking the table then sometimes it is better execute within this window. Frequently though we are reactive to new SQL statements or a new performance issue and an emergency index is required to reduce load on the master in order to improve the response time. To pt-osc or not to pt-osc? As mentioned, pt-online-schema-change is a fixture in our workflow. It’s usually the right way to go but we still have occasions where pt-online-schema-change cannot be used, for example; when a table already uses triggers. It’s an important to remind ourselves of the the steps that pt-online-schema-change traverses to complete it’s job. Lets look at the source code to identify these; [moore@localhost]$ egrep 'Step' pt-online-schema-change # Step 1: Create the new table. # Step 2: Alter the new, empty table. This should be very quick, # Step 3: Create the triggers to capture changes on the original table and <--(metadata lock) # Step 4: Copy rows. # Step 5: Rename tables: orig -> old, new -> orig <--(metadata lock) # Step 6: Update foreign key constraints if there are child tables. # Step 7: Drop the old table. [moore@localhost]$egrep'Step'pt-online-schema-change # Step 1: Create the new table. # Step 2: Alter the new, empty table. This should be very quick, # Step 3: Create the triggers to capture changes on the original table and <--(metadata lock) # Step 4: Copy rows. # Step 5: Rename tables: orig -> old, new -> orig <--(metadata lock) # Step 6: Update foreign key constraints if there are child tables. # Step 7: Drop the old table. I pick out steps 3 and 5 from above to highlight a source of a source of potential downtime due to locks, but step 6 is also an area for concern since foreign keys can have nested actions and should be considered when planning these actions to avoid related tables from being rebuilt with a direct alter implicitly. There are several ways to approach a table with referential integrity constraints and they are detailed within the pt-osc documentation a good preparation step is to review the structure of your table including the constraints and how the ripples of the change can affect the tables around it. Recently we were alerted to an incident after a client with a highly concurrent and highly transactional workload ran a standard pt-online-schema-change script over a large table. This appeared normal to them and a few hours later our pager man was notified that this client was experiencing max_connections limit reached. So what was going on? When pt-online-schema-change reached step 5 it tried to acquire a metadata lock to rename the the original and the shadow table, however this wasn’t immediately granted due to open transactions and thus threads began to queue behind the RENAME command. The actual effect this had on the client’s application was downtime. No new connections could be made and all existing threads were waiting behind the RENAME command. Metadata locks Introduced in 5.5.3 at server level. When a transaction starts it will acquire a metadata lock (independent of storage engine) on all tables it uses and then releases them when it’s finished it’s work. This ensures that nothing can alter the table definition whilst a transaction is open. With some foresight and planning we can avoid these situations with non-default pt-osc options, namely –nodrop-new-table and –no-swap-tables. This combination leaves both the shadow table and the triggers inplace so that we can instigate an atomic RENAME when load permits. EDIT: as of percona-toolkit version 2.2 we have a new variable –tries which in conjunction with –set-vars has been deployed to cover this scenario where various pt-osc operations could block waiting for a metadata lock. The default behaviour of pt-osc (–set-vars) is to set the following session variables when it connects to the server; wait_timeout=10000 innodb_lock_wait_timeout=1 lock_wait_timeout=60 when using –tries we can granularly identify the operation, try count and the wait interval between tries. This combination will ensure that pt-osc will kill it’s own waiting session in good time to avoid the thread pileup and provide us with a loop to attempt to acquire our metadata lock for triggers|rename|fk management; –tries swap_tables:5:0.5,drop_triggers:5:0.5 The documentation is here http://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html#cmdoption-pt-online-schema-change–tries This illustrates that even with a tool like pt-online-schema-change it is important to understand the caveats presented with the solution you think is most adequate. To help decide the direction to take use the flow chart to ensure you’re taking into account some of the caveats of the MySQL schema change. Be sure to read up on the recommended outcome though as there are uncharted areas such as disk space, IO load that are not featured on the diagram. Choosing the right DDL option Ensure you know what effect ALTER TABLE will have on your platform and pick the right method to suit your uptime. Sometimes that means delaying the change until a period of lighter use or utilising a tool that will avoid holding a table locked for the duration of the operation. A direct ALTER is sometimes the answer like when you have triggers installed on a table. – In most cases pt-osc is exactly what we need – In many cases pt-osc is needed but the way in which it’s used needs tweaking – In few cases pt-osc isn’t the right tool/method and we need to consider native blocking ALTER or using failovers to juggle the change into place on all hosts in the replica cluster. If you want to learn more about avoiding avoidable downtime please tune into my webinar Wednesday, November 19 at 10 a.m. PST. It’s titled “Tips from the Trenches: A Guide to Preventing Downtime for the Over-Extended DBA.” Register now!(If you miss it, don’t worry: Catch the recording and download the slides from that same registration page.)

In this article I will attempt to run the Mule ESB community edition in Docker in order to see whether it is feasible without any greater inconvenience. My goal is to be able to use Docker both when testing as well as in a production environment in order to gain better control over the environment and to separate different types of environments. I imagine that most of the Docker-related information can be applied to other applications – I have used Mule since it is what I usually work with. The conclusion I have made after having completed my experiments is that it is possible to run Mule ESB in Docker without any inconvenience. In addition, Docker will indeed allow me to have better control over the different environments and also allow me to separate them as I find appropriate. Finally, I just want to mention that I have used Docker in an Ubuntu environment. I have not attempted any of the exercises in Docker running on Windows or Mac OS X. Docker Briefly In short, Docker allows for creating of images that serve as blueprints for containers. A Docker container is an instance of a Docker image in the same way a Java object is an instance of a Java class. FROM codingtony/java MAINTAINER tony(dot)bussieres(at)ticksmith(dot)com RUN wget https://repository.mulesoft.org/nexus/content/repositories/releases/org/mule/distributions/mule-standalone/3.5.0/mule-standalone-3.5.0.tar.gz RUN cd /opt && tar xvzf ~/mule-standalone-3.5.0.tar.gz RUN echo "4a94356f7401ac8be30a992a414ca9b9 /mule-standalone-3.5.0.tar.gz" | md5sum -c RUN rm ~/mule-standalone-3.5.0.tar.gz RUN ln -s /opt/mule-standalone-3.5.0 /opt/mule CMD [ "/opt/mule/bin/mule" ] The resource isolation features of Linux are used to create Docker containers, which are more lightweight than virtual machines and are separated from the environment in which Docker runs, the host. Using Docker an image can be created that, every time it is started has a known state. In order to remove any doubts about whether the environment has been altered in any way, the container can be stopped and a new container started. I can even run multiple Docker containers on one and the same computer to simulate a multi-server production environment. Applications can also be run in their own Docker containers, as shown in this figure. Three Docker containers, each containing a specific application, running in one host. A more detailed introduction to Docker is available here. The main entry point to the Docker documentation can be found here. Motivation Some of the motivations I have for using Docker in both testing and production environments are: The environment in which I test my application should be as similar as the final deployment environment as possible, if not identical. Making the deployment environment easy to scale up and down. If it is easy to start a new processing node when need arise and stop it if it is no longer used, I will be able to adapt to changes rather quickly and thus reduce errors caused by, for instance, load peaks. Maintain an increased number of nodes to which applications can be deployed. Instead of running one instance of some kind of application server, Mule ESB in my case, on a computer, I want multiple instances that are partitioned, for instance, according to importance. High-priority applications run on one separate instance, which have higher priority both as far as resources (CPU, memory, disk etc) are concerned but also as far as support is concerned. Applications which are less critical run on another instance. Enable quick replacement of instances in the deployment environment. Reasons for having to replace instances may be hardware failure etc. Better control over the contents of the different environments. The concept of an environment that, at any time, may be disposed (and restarted) discourages hacks in the environment, which are usually poorly documented and sometimes difficult to trace. Using Docker, I need to change the appropriate Docker image if I want to make changes to some application environment. The Docker image file, commonly known as Dockerfile, can be checked into any ordinary revision control system, such as Git, Subversion etc, making changes reversible and traceable. Automate the creation of a testing environment. An example could be a nightly job that runs on my build server which creates a test environment, deploys one or more applications to it and then performs tests, such as load-testing. Prerequisites To get the best possible experience when running Docker, I run it under Ubuntu. According to the current documentation, Docker is supported under the following versions of Ubuntu: 12.04 LTS (64-bit) 13.04 (64-bit) 13.10 (64-bit) 14.04 (64-bit) Against my usual conservative self, I chose Ubuntu 14.10, which at the time of writing this article is the latest version. While I haven’t run into any issues, I cannot promise anything regarding compatibility with Docker as far as this version of Ubuntu is concerned. Installing Docker Before we install anything, those who have the Docker version from the Ubuntu repository should remove this version before installing a newer version of Docker, since the Ubuntu repository does not contain the most recent version and the package does not have the same name as the Docker package we will install: sudo apt-get remove docker.io The simplest way to install Docker is to use an installation script made available at the Docker website: curl -sSL https://get.docker.com/ubuntu/ | sudo sh If you are not running Ubuntu or if you do not want to use the above way of installing Docker, please refer to this page containing instructions on how to install Docker on various platforms. To verify the Docker installation, open a terminal window and enter: sudo docker version Output similar to the following should appear: Client version: 1.4.1 Client API version: 1.16 Go version (client): go1.3.3 Git commit (client): 5bc2ff8 OS/Arch (client): linux/amd64 Server version: 1.4.1 Server API version: 1.16 Go version (server): go1.3.3 Git commit (server): 5bc2ff8 We are now ready to start a Mule instance in Docker. Running Mule in Docker One of the advantages with Docker is that there is a large repository of Docker images that are ready to be used, and even extended if one so wishes. ThisDocker image is the one that I will use in this article. It is well documented, there is a source repository and it contains a recent version of the Mule ESB Community Edition. Some additional details on the Docker image: Ubuntu 14.04. Oracle JavaSE 1.7.0_65. This version will change as the PPA containing the package is updated. Mule ESB CE 3.5.0 Note that the image may change at any time and the specifications above may have changed. If you intend to use Docker in your organization, I would suspect that the best alternative is to create your own Docker images that are totally under your control. The Docker image repository is an excellent source of inspiration and aid even in this case. Starting a Docker Container To start a Docker container using this image, open a terminal window and write: sudo docker run codingtony/mule The first time an image is used it needs to be downloaded and created. This usually takes quite some time, so I suggest a short break here – perhaps for a cup of coffee or tea. If you just want to download an image without starting it, exchange the Docker command “run” with “pull”. Once the container is started, you will see some output to the console. If you are familiar with Mule, you will recognize the log output: MULE_HOME is set to /opt/mule-standalone-3.5.0 Running in console (foreground) mode by default, use Ctrl-C to exit... MULE_HOME is set to /opt/mule-standalone-3.5.0 Running Mule... --> Wrapper Started as Console Launching a JVM... Starting the Mule Container... Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved. INFO 2015-01-05 04:41:42,302 [WrapperListener_start_runner] org.mule.module.launcher.MuleContainer: ********************************************************************** * Mule ESB and Integration Platform * * Version: 3.5.0 Build: ff1df1f3 * * MuleSoft, Inc. * * For more information go to http://www.mulesoft.org * * * * Server started: 1/5/15 4:41 AM * * JDK: 1.7.0_65 (mixed mode) * * OS: Linux (3.16.0-28-generic, amd64) * * Host: f95698cfb796 (172.17.0.2) * ********************************************************************** Note that: In the text-box containing information about the Mule ESB and Integration Platform, there is a row which starts with “Host:”. The hexadecimal digit that follows is the Docker container id and the IP-address is the external IP-address of the Docker container in which Mule is running. Before we do anything with the Mule instance running in Docker, let’s take a look at Docker containers. Docker Containers We can verify that there is a Docker container running by opening another terminal window, or a tab in the first terminal window, and running the command: sudo docker ps As a result, you will see output similar to the following (I have edited the output in order for the columns to be aligned with the column titles): CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f95698cfb796 codingtony/mule:latest "/opt/mule/bin/mule" 7 min ago Up 7 min jolly_hopper From this output we can see that: The ID of the container is f95698cfb796. This ID can be used when performing operations on the container, such as stopping it, restarting it etc. The name of the image used to created the container. The command that is currently executing. If we look at the Dockerfile for the image, we can see that the last line in this file is: CMD [ “/opt/mule/bin/mule” ] This is the command that is executed whenever an instance of the Docker image is launched and it matches what we see in the COMMAND column for the Docker container. The CREATED column shows how much time has passed since the container was created. The STATUS column shows the current status of the image. When you have used Docker for a while, you can view all the containers using: sudo docker ps -a This will show you containers that are not running, in addition to the running ones. Containers that are not running can be restarted. The PORTS column shows any port mappings for the container. More about port mappings later. Finally, the NAMES column contain a more human-friendly container name. This container name can be used in the same way as the container id. Docker containers will consume disk-space and if you want to determine how much disk-space each of the containers on your computer use, issue the following command: sudo docker ps -a -s An additional column, SIZE, will be shown and in this column I see that my Mule container consumes 41,76kB. Note that this is in addition to the disk-space consumed by the Docker image. This number will grow if you use the container under a longer period of time, as the container retains any files written to disk. To completely remove a stopped Docker container, find the id or name of the container and use the command: sudo docker rm [container id or name here] Before going further, let’s stop the running container and remove it: sudo docker stop [container id or name here] sudo docker rm [container id or name here] Files and Docker Containers So far we have managed to start a Mule instance running inside a Docker container, but there were no Mule applications deployed to it and the logs that were generated were only visible in the terminal window. I want to be able to deploy my applications to the Mule instance and examine the logs in a convenient way. In this section I will show how to: Share one or more directories in the host file-system with a Docker container. Access the files in a Docker container from the host. As the first step in looking at sharing directories between the host operating system and a Docker container, we are going to look at Mule logs. As part of this exercise we also set up the directories in the host operating system that are going to be shared with the Docker container. In your home directory, create a directory named “mule-root”. In the “mule-root” directory, create three directories named “apps”, “conf” and “logs”. Download the Mule CE 3.5.0 standalone distribution from this link. From the Mule CE 3.5.0 distribution, copy the files in the “apps” directory to the “mule-root/apps” directory you just created. From the Mule CE 3.5.0 distribution, copy the files in the “conf” directory to the “mule-root/conf” directory you created. The resulting file- and directory-structure should look like this (shown using the tree command): ~/mule-root/ ├── apps │ └── default │ └── mule-config.xml ├── conf │ ├── log4j.properties │ ├── tls-default.conf │ ├── tls-fips140-2.conf │ ├── wrapper-additional.conf │ └── wrapper.conf └── logs Edit the log4j.properties file in the “mule-root/conf” directory and set the log-level on the last line in the file to “DEBUG”. This modification has nothing to do with sharing directories, but is in order for us to be able to see some more output from Mule when we run it later. The last two lines should now look like this: # Mule classes log4j.logger.org.mule=DEBUG Binding Volumes We are now ready to launch a new Docker container and when we do, we will tell Docker to map three directories in the Docker container to three directories in the host operating system. Three directories in a Docker container bound to three directories in the host. Launch the Docker container with the command below. The -v option tells Docker that we want to make the contents of a directory in the host available at a certain path in the Docker container file-system. The -d option runs the container in the background and the terminal prompt will be available as soon as the id of the newly launched Docker container has been printed. sudo docker run -d -v ~/mule-root/apps:/opt/mule/apps -v ~/mule-root/conf:/opt/mule/conf -v ~/mule-root/logs:/opt/mule/logs codingtony/mule Examine the “mule-root” directory and its subdirectories in the host, which should now look like below. The files on the highlighted rows have been created by Mule. mule-root/ ├── apps │ ├── default │ │ └── mule-config.xml │ └── default-anchor.txt ├── conf │ ├── log4j.properties │ ├── tls-default.conf │ ├── tls-fips140-2.conf │ ├── wrapper-additional.conf │ └── wrapper.conf └── logs ├── mule-app-default.log ├── mule-domain-default.log └── mule.log Examine the “mule.log” file using the command “tail -f ~/mule-root/logs/mule.log”. There should be periodic output written to the log file similar to the following: DEBUG 2015-01-05 12:05:37,216 [Mule.app.deployer.monitor.1.thread.1] org.mule.module.launcher.DeploymentDirectoryWatcher: Checking for changes... DEBUG 2015-01-05 12:05:37,216 [Mule.app.deployer.monitor.1.thread.1] org.mule.module.launcher.DeploymentDirectoryWatcher: Current anchors: default-anchor.txt DEBUG 2015-01-05 12:05:37,216 [Mule.app.deployer.monitor.1.thread.1] org.mule.module.launcher.DeploymentDirectoryWatcher: Deleted anchors: Stop and remove the container: sudo docker stop [container id or name here] sudo docker rm [container id or name here] Direct Access to Docker Container Files When running Docker under the Ubuntu OS it is also possible to access the file-system of a Docker container from the host file-system. It may be possible to do this under other operating systems too, but I haven’t had the opportunity to test this. This technique may come in handy during development or testing with Docker containers for which you haven’t bound any volumes. Note! If given the choice to use either volume binding, as seen above, or direct access to container files as we will look at in this section for something more than a temporary file access, I would chose to use volume binding. Direct access to Docker container files relies on implementation details that I suspect may change in future versions of Docker if the developers find it suitable. With all that said, lets get the action started: Start a new Docker container: sudo docker run -d codingtony/mule Find the id of the newly launched Docker container: sudo docker ps Examine low-level information about the newly launched Docker container: sudo docker inspect [container id or name here] Output similar to this will be printed to the console (portions removed to conserve space): [{ "AppArmorProfile": "", "Args": [], "Config": { ... }, "Created": "2015-01-12T07:58:47.913905369Z", "Driver": "aufs", "ExecDriver": "native-0.2", "HostConfig": { ... }, "HostnamePath": "/var/lib/docker/containers/68b40def7ad6a7f819bd654d5627ad1c3a0f40c84e0fb0f875760f1bd6790eef/hostname", "HostsPath": "/var/lib/docker/containers/68b40def7ad6a7f819bd654d5627ad1c3a0f40c84e0fb0f875760f1bd6790eef/hosts", "Id": "68b40def7ad6a7f819bd654d5627ad1c3a0f40c84e0fb0f875760f1bd6790eef", "Image": "bcd0f37d48d4501ad64bae941d95446b157a6f15e31251e26918dbac542d731f", "MountLabel": "", "Name": "/thirsty_darwin", "NetworkSettings": { ... }, "Path": "/opt/mule/bin/mule", "ProcessLabel": "", "ResolvConfPath": "/var/lib/docker/containers/68b40def7ad6a7f819bd654d5627ad1c3a0f40c84e0fb0f875760f1bd6790eef/resolv.conf", "State": { ... }, "Volumes": {}, "VolumesRW": {} }] Locate the “Driver” node (highlighted in the above output) and ensure that its value is “aufs”. If it is not, you may need to modify the directory paths below replacing “aufs” with the value of this node. Personally I have only seen the “aufs” value at this node so anything else is uncharted territory to me. Copy the long hexadecimal value that can be found at the “Id” node (also highlighted in the above output). This is the long id of the Docker container. In a terminal window, issue the following command, inserting the long id of your container where noted: sudo ls -al /var/lib/docker/aufs/mnt/[long container id here] You are now looking at the root of the volume used by the Docker container you just launched. In the same terminal window, issue the following command: sudo ls -al /var/lib/docker/aufs/mnt/[long container id here]/opt The output from this command should look like this: total 12 drwxr-xr-x 4 root root 4096 jan 12 15:58 . drwxr-xr-x 75 root root 4096 jan 12 15:58 .. lrwxrwxrwx 1 root root 26 aug 10 04:19 mule -> /opt/mule-standalone-3.5.0 drwxr-xr-x 17 409 409 4096 jan 12 15:58 mule-standalone-3.5.0 Examine this line in the Dockerfile:RUN ln -s /opt/mule-standalone-3.5.0 /opt/muleWe see that a symbolic link is created and that the directory name and the name of the symbolic link matches the output we saw earlier. This matches the directory output in the previous step. To examine the Mule log file that we looked at when binding volumes earlier, use the following command: sudo cat /var/lib/docker/aufs/mnt/[long container id here]/opt/mule-standalone-3.5.0/logs/mule.log Next we create a new file in the Docker container using vi: sudo vi /var/lib/docker/aufs/mnt/[long container id here]/opt/mule-standalone-3.5.0/test.txt Enter some text into the new file by first pressing i and the type the text. When you are finished entering the text, press the Escape key and write the file to disk by typing the characters “:wq” without quotes. This writes the new contents of the file to disk and quits the editor. Leave the Docker container running after you are finished. In the next section, we are going to look at the file we just created from inside the Docker container. We have seen that we can examine the file system of a Docker container without binding volumes. It is also possible to copy or move files from the host file-system to the container’s file system using the regular commands. Root privileges are required both when examining and writing to the Docker container’s file system. Entering a Docker Container In order to verify that the file we just created in the host was indeed written to the Docker container, we are going to start a bash shell in the running Docker container and examine the location where the new file is expected to be located and the contents of the file. In the process we will see how we can execute commands in a Docker container from the host. Issue the command below in a terminal window. The exec Docker command is used to run a command, bash in this case, in a running Docker container. The -i flags tell Docker to keep the input stream open while the command is being executed. In this example, it allows us to enter commands into the bash shell running inside the Docker container. The -t flag cause Docker to allocate a text terminal to which the output from the command execution is printed. sudo docker exec -i -t [container id or name here] bash Note the prompt, which should change to [user]@[Docker container id]. In my case it looks like this: root@3ea374a280da:/# Go to the Mule installation directory using this command: cd /opt/mule-standalone-3.5.0/ Examine the contents of the directory: ls -al Among the other files, you should see the “test.txt” file: -rw-r--r-- 1 root root 53 Jan 14 03:19 test.txt Examine the contents of the “text.txt” file. The contents of the file should match what you entered earlier. cat text.txt Exit to the host OS: exit Stop and remove the container: sudo docker stop [container id or name here] sudo docker rm [container id or name here] We have seen that we can execute commands in a running Docker container. In this particular example, we used it to execute the bash shell and examine a file. I draw the conclusion that I should be able to set up a Docker image that contains a very controlled environment for some type of test and then create a container from that image and start the test from the host. Deploying a Mule Application In this section we will look at deploying a Mule application to an instance of the Mule ESB running in a Docker container. We will use volume binding, that we looked at in the section on files and Docker containers, to share directories in the host with the Docker container in order to make it easy to deploy applications, modify running applications, examine logs etc. Preparations Before deploying the application, we need to make some preparations: First of all, we restore the original log-level that we changed earlier. In this example, there will be log output when the applications we will deploy is run and we can limit the log generated by Mule. Edit the log4j.properties file in the “mule-root/conf” directory in the host and set the log-level on the last line in the file back to “INFO” and add one line, as in the listing below. The last three lines should now look like this: # Mule classes log4j.logger.org.mule=INFO log4j.logger.org.mule.tck.functional=DEBUG Next, we create the Mule application which we will deploy to the Mule ESB running in Docker: In some directory, create a file named “mule-deploy.properties” with the following contents: redeployment.enabled=true encoding=UTF-8 domain=default config.resources=HelloWorld.xml In the same directory create a file named “HelloWorld.xml”. This file contains the Mule configuration for our example application: Create a zip-archive named “mule-hello.zip” containing the two files created above: zip mule-hello.zip mule-deploy.properties HelloWorld.xml Deploy the Mule Application Before you start the Docker container in which the Mule EBS will run, make sure that you have created and prepared the directories in the host as described in the section Files and Docker Containers above. Start a new Mule Docker container using the command that we used when binding volumes: sudo docker run -d -v ~/mule-root/apps:/opt/mule/apps -v ~/mule-root/conf:/opt/mule/conf -v ~/mule-root/logs:/opt/mule/logs codingtony/mule As before, the -v option tells Docker to bind three directories in the host to three locations in the Docker container’s file system. Find the IP-address of the Docker container: sudo docker inspect [container id or name here] | grep IPAddress In my case, I see the following line which reveals the IP-address of the Docker container: “IPAddress”: “172.0.17.2”, Open a terminal window or tab and examine the Mule log. Leave this window or tab open during the exercise, in order to be able to verify the output from Mule. tail -f ~/mule-root/logs/mule.log Copy the zip-archive “mule-hello.zip” created earlier to the host directory ~/mule-root/apps/. Verify that the application has been deployed without errors in the Mule log: ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + Started app 'mule-hello' + ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Leave the Docker container running after you are finished. In the next section we will look at how to access endpoints exposed by applications running in Docker containers. By binding directories in the host thus making them available in the Docker container, it becomes very simple to deploy Mule applications to an instance of Mule ESB running in a Docker container. I am considering this setup for a production environment as well, since it will enable me to perform backups of the directories containing Mule applications and configuration without having to access the Docker container’s file system. It is also in accord with the idea that a Docker container should be able to be quickly and easily restarted, which I feel it would not be if I had to deploy a number of Mule applications to it in order to recreate its previous state. Accessing Endpoints We now know that we can run the Mule ESB in a Docker container, we can deploy applications and examine the logs quite easily but one final, very important question remains to be answered; how to access endpoints exposed by applications running in a Docker container. This section assumes that the Mule application we deployed to Mule in the previous section is still running. In the host, open a web-browser and issue a request to the Docker container’s IP-address at port 8181. In my case, the URL is http://172.17.0.2:8181 Alternatively use the curl command in a terminal window. In my case I would write: curl 172.17.0.2:8181 The result should be a greeting in the following format: Hello World! It is now: 2015-01-14T07:39:03.942Z In addition, you should be able to see that a message was received in the Mule log. Now try the URL http://localhost:8181 You will get a message saying that the connection was refused, provided that you do not already have a service listening at that port. If you have another computer available that is connected to the same network as the host computer running Ubuntu, do the following: – Find the IP-address of the Ubuntu host computer using the ifconfigcommand. – In a web-browser on the other computer, try accessing port 8181 at the IP-address of the Ubuntu host computer. Again you will get a message saying that the connection was refused. Stop and remove the container: sudo docker stop [container id or name here] sudo docker rm [container id or name here] Without any particular measures taken, we see that we can access a service exposed in a Docker container from the Docker host but we did not succeed in accessing the service from another computer. To make a service exposed in a Docker container reachable from outside of the host, we need to tell Docker to publish a port from the Docker container to a port in the host using the -p flag: Launch a new Docker container using the following command: sudo docker run -d -p 8181:8181 -v ~/mule-root/apps:/opt/mule/apps -v ~/mule-root/conf:/opt/mule/conf -v ~/mule-root/logs:/opt/mule/logs codingtony/mule The added flag -p 8181:8181 makes the service exposed at port 8181 in the Docker container available at port 8181 in the host. Try accessing the URL http://localhost:8181 from a web-browser on the host computer.The result should be a greeting of the form we have seen earlier. Try accessing port 8181 at the IP-address of the Ubuntu host computer from another computer.This should also result in a greeting message. Stop and remove the container: sudo docker stop [container id or name here] sudo docker rm [container id or name here] Using the -p flag, we have seen that we can expose a service in a Docker container so that it becomes accessible from outside of the host computer. However, we also see that this information need to be supplied at the time of launching the Docker container. The conclusions that I draw from this is that: I can test and develop against a Mule ESB instance running in a Docker container without having to publish any ports, provided that my development computer is the Docker host computer. In a production environment or any other environment that need to expose services running in a Docker container to “the outside world” and where services will be added over time, I would consider deploying an Apache HTTP Server or NGINX on the Docker host computer and use it to proxy the services that are to be exposed. This way I can avoid re-launching the Docker container each time a new service is added and I can even (temporarily) redirect the proxy to some other computer if I need to perform some maintenance. Is There More? Of course! This article should only be considered an introduction and I am just a beginner with Docker. I hope I will have the time and inspiration to write more about Docker as I learn more.

In this post you can see an approach for conditionally enabling/disabling a set of checkboxes. For this we can use the ng-disabled directive and some CSS clases of typeclassName-true and className-false: ENABLE/DISABLE CHECKBOXES USING ANGULAR JS Select the maximum prize money: Select one prize money{{item.prizemoney} {{item.name}

An increasing number of systems are being built to handle the Volume, Velocity and Variety of Big Data, and hopefully help gain new insights and make better business decisions. Here, we will look at ways to deal with Big Data’s Volume and Velocity simultaneously, within a single architecture solution. Volume + Velocity Apache Hadoop provides both reliable storage (HDFS) and a processing system (MapReduce) for large data sets across clusters of computers. MapReduce is a batch query processor that is targeted at long-running background processes. Hadoop can handle Volume. But to handle Velocity, we need real-time processing tools that can compensate for the high-latency of batch systems, and serve the most recent data continuously, as new data arrives and older data is progressively integrated into the batch framework. Therefore we need both batch and real-time to run in parallel, and add a real-time computational system (e.g. Apache Storm) to our batch framework. This architectural combination of batch and real-time computation is referred to as a Lambda Architecture (λ). Generic Lambda λ has three layers: The Batch Layer manages the master data and precomputes the batch views The Speed Layer serves recent data only and increments the real-time views The Serving Layer is responsible for indexing and exposing the views so that they can be queried. The three layers are outlined in the below diagram along with a sample choice of technology stacks: Incoming data is dispatched to both Batch and Speed layers for processing. At the other end, queries are answered by merging both batch and real-time views. Note that real-time views are transient by nature and their data is discarded (making room for newer data) once propagated through the Batch and Serving layers. Most of the complexity is pushed onto the much smaller Speed layer where the results are only temporary, a process known as “complexity isolation“. We are indeed isolating the complexity of concurrent data updates in a layer that is regularly purged and kept small in size. λ is technology agnostic. The data pipeline is broken down into layers with clear demarcation of responsibilities, and at each layer, we can choose from a number of technologies. The Speed layer for instance could use either Apache Storm, or Apache Spark Streaming, or Spring “XD” ( eXtreme Data) etc. How do we recover from mistakes in λ ? Basically, we recompute the views. If that takes too long, we just revert to the previous, non-corrupted versions of our data. We can do that because of data immutability in the master dataset: data is never updated, only appended to (time-based ordering). The system is therefore Human Fault-Tolerant: if we write bad data, we can just remove that data altogether and recompute. Unified Lambda The downside of λ is its inherent complexity. Keeping in sync two already complex distributed systems is quite an implementation and maintenance challenge. People have started to look for simpler alternatives that would bring just about the same benefits and handle the full problem set. There are basically three approaches: 1) Adopt a pure streaming approach, and use a flexible framework such as Apache Samza to provide some type of batch processing. Although its distributed streaming layer is pluggable, Samza typically relies on Apache Kafka. Samza’s streams are replayable, ordered partitions. Samza can be configured for batching, i.e. consume several messages from the same stream partition in sequence. 2) Take the opposite approach, and choose a flexible Batch framework that would also allow micro-batches, small enough to be close to real-time, with Apache Spark/Spark Streaming or Storm’s Trident. Spark streaming is essentially a sequence of small batch processes that can reach latency as low as one second.Trident is a high-level abstraction on top of Storm that can process streams as small batches as well as do batch aggregation. 3) Use a technology stack already combining batch and real-time, such as Spring “XD”, Summingbird or Lambdoop. Summingbird (“Streaming MapReduce”) is a hybrid system where both batch/real-time workflows can be run at the same time and the results merged automatically.The Speed layer runs on Storm and the Batch layer on Hadoop, Lambdoop (Lambda-Hadoop, with HBase, Storm and Redis) also combines batch/real-time by offering a single API for both processing paradigms: The integrated approach (unified λ) seeks to handle Big Data’s Volume and Velocity by featuring a hybrid computation model, where both batch and real-time data processing are combined transparently. And with a unified framework, there would be only one system to learn, and one system to maintain.

In this tip you can see how to use the Angular Websocket module for connecting client applications to servers.

Posted by Gilad F on Back& Blog. Current approaches to web development rely upon having two kinds of intelligence built into your application – business intelligence in the server, and presentation intelligence on the client side. This institutes a clear delineation in responsibilities, which is often desirable from an architectural standpoint. However, this approach does have some drawbacks. Processing time for business logic, for example, is centralized on the server. This can introduce bottlenecks in the application’s performance, or add complexity when it comes to cross-server communication. For smaller applications that nonetheless have a large user base, this can often be the single greatest performance concern – the time spent computing solutions by the server. One way this can be offset is through the use of Object-Relational Mapping, or ORM. Below we’ll look at the concept of ORM, and how creating an ORM system in Angular can help make your application smarter. What is an ORM? Simply put, Object-Relational Mapping is the concept of creating representations of your underlying data that know how to manage themselves. Most web applications boil down to four basic actions, known as the “CRUD” approach – Create a record, Retrieve records, Update a record, or Delete a record. With an ORM, you simply encapsulate each of these functions within a class that represents a given record in the database. In essence, the objects you create to represent your data on the front end also know how to manipulate that data on the back end. Why Use an ORM? The primary benefit of an ORM is that it hides a lot of the functional complexity of database integration behind an established API. Communication with the database to implement each of the CRUD methods can be complex, but once it’s been accomplished for one model it can be easily ported to all of the other models in your system. An ORM focuses on hiding as much of this code as possible, allowing your models to care only about how they are represented – and how they interact with other elements in the system. A series of calls to establish a connection to the database, for example, becomes a single call to a method named “Save” on the model instance. This also allows you to centralize your database code, giving you only one location where you need to look for database-related bugs instead of having to search a complex code base for different custom data communication handlers. Why Use an ORM in Angular? While the JavaScript stack is particularly performant when compared to more heavyweight offerings such as Rails and Django, it still faces the issues common to the standard web application architecture – the server has the potential to be a bottleneck, handling the incoming traffic from a number of locations. By focusing your development efforts to create a pure CRUD API in your server, and developing a rudimentary ORM in Angular, you can offload a lot of that processing load to the client machines – in essence parallelizing the process at the expense of increased network communication. This allows you to reduce the overall dependence of your application on the server, making the server a “thin” client that simply updates the database based upon the API calls issued by the client. After a certain point, your back-end can be outsourced completely to an external provider that specializes in providing this type of access – such as Backand – allowing you to completely offload scalability and security concerns. In essence, it allows you to focus on your application as opposed to focusing on the attendant resources. Conclusion Object-Relational Mapping is a powerful paradigm that eases communication with a database for the basic CRUD activities associated with web applications. As most existing web development environments focus on implementing ORM on the server side, this can result in performance and communication bottlenecks – not to mention increased infrastructure costs. By offloading some of these ORM tasks to AngularJS, you can parallelize many of these tasks and reduce overall server load, in some cases obviating the need for the server entirely. If your application is facing a bloated back-end communication pattern, it might be worth your time to look at working towards implementation of a client-side ORM system. Build your Angular app and connect it to any database with Backand today. – Get started now.translate in hindi

as i said in resolving the conflict between software architecture and code , my focus for this year is representing a software architecture model as code. in simple sketches for diagramming your software architecture , i showed an example system context diagram for my techtribes.je website. it's a simple diagram that shows techtribes.je in the middle, surrounded by the key types of users and system dependencies. it's your typical "big picture" view. this diagram was created using omnigraffle (think microsoft visio for mac os x) and it's exactly that - a static diagram that needs to be manually kept up to date. instead, wouldn't it be great if this diagram was based upon a model that we could better version control, collaborate on and visualize? if you're not sure what i mean by a "model", take a look at models, sketches and everything in between . this is basically what the aim of structurizr is. it's a way to describe a software architecture model as code, and then visualize it in a simple way. the structurizr java library is available on github and you can download a prebuilt binary . just as a warning, this is very much a work in progress and so don't be surprised if things change! here's some java code to recreate the techtribes.je system context diagram. package com.structurizr.example; import com.structurizr.io.json.jsonwriter; import com.structurizr.model.location; import com.structurizr.model.model; import com.structurizr.model.person; import com.structurizr.model.softwaresystem; import com.structurizr.view.systemcontextview; import com.structurizr.view.viewset; import java.io.stringwriter; /** * this is a model of the system context for the techtribes.je system, * the code for which can be found at https://github.com/techtribesje/techtribesje */ public class techtribessystemcontext { public static void main(string[] args) throws exception { // create a model and the software system we want to describe model model = new model("techtribes.je", "this is a model of the system context for the techtribes.je system, the code for which can be found at https://github.com/techtribesje/techtribesje"); softwaresystem techtribes = model.addsoftwaresystem(location.internal, "techtribes.je", "techtribes.je is the only way to keep up to date with the it, tech and digital sector in jersey and guernsey, channel islands"); // create the various types of people (roles) that use the software system person anonymoususer = model.addperson(location.external, "anonymous user", "anybody on the web."); anonymoususer.uses(techtribes, "view people, tribes (businesses, communities and interest groups), content, events, jobs, etc from the local tech, digital and it sector."); person authenticateduser = model.addperson(location.external, "aggregated user", "a user or business with content that is aggregated into the website."); authenticateduser.uses(techtribes, "manage user profile and tribe membership."); person adminuser = model.addperson(location.external, "administration user", "a system administration user."); adminuser.uses(techtribes, "add people, add tribes and manage tribe membership."); // create the various software systems that techtribes.je has a dependency on softwaresystem twitter = model.addsoftwaresystem(location.external, "twitter", "twitter.com"); techtribes.uses(twitter, "gets profile information and tweets from."); softwaresystem github = model.addsoftwaresystem(location.external, "github", "github.com"); techtribes.uses(github, "gets information about public code repositories from."); softwaresystem blogs = model.addsoftwaresystem(location.external, "blogs", "rss and atom feeds"); techtribes.uses(blogs, "gets content using rss and atom feeds from."); // now create the system context view based upon the model viewset viewset = new viewset(model); systemcontextview contextview = viewset.createcontextview(techtribes); contextview.addallsoftwaresystems(); contextview.addallpeople(); // and output the model and view to json (so that we can render it using structurizr.com) jsonwriter jsonwriter = new jsonwriter(true); stringwriter stringwriter = new stringwriter(); jsonwriter.write(viewset, stringwriter); system.out.println(stringwriter.tostring()); } } executing this code creates this json , which you can then copy and paste into the try it page of structurizr. the result (if you move the boxes around) is something like this. don't worry, there will eventually be an api for uploading software architecture models and the diagrams will get some styling, but it proves the concept. what we have then is an api that implements the various levels in my c4 software architecture model, with a simple browser-based rendering tool. hopefully that's a nice simple introduction of how to represent a software architecture model as code, and gives you a flavour for the sort of direction i'm taking it. having the software architecture as code provides some interesting opportunities that you don't get with static diagrams from visio, etc and the ability to keep the models up to date automatically by scanning the codebase is what i find particularly exciting. if you have any thoughts on this, please do drop me a note.

In this article I am going to present a solution about sending SMS messages in Java for those developers and marketers who think – like me – that SMS is not dead.

I’ve lost count of the number of times I’ve seen code which fail-fast validates the state of something, using an approach like public class PersonValidator { public boolean validate(Person person) { boolean valid = person != null; if (valid) valid = person.givenName != null; if (valid) valid = person.familyName != null; if (valid) valid = person.age != null; if (valid) valid = person.gender != null; // ...and many more } } It works, but it’s a brute force approach that’s filled with repetition due to the valid check. If your code style enforces braces for if statements (+1 for that), your method is also three times longer and growing every time a new check is added to the validator. Using Java 8’s new stream API, we can improve this by taking the guard condition of if (valid) and making a generic validator that handles the plumbing for you. import java.util.LinkedList; import java.util.List; import java.util.function.Predicate; public class GenericValidator implements Predicate { private final List> validators = new LinkedList<>(); public GenericValidator(List> validators) { this.validators.addAll(validators); } @Override public boolean test(final T toValidate) { return validators.parallelStream() .allMatch(predicate -> predicate.test(toValidate)); } } Using this, we can rewrite the Person validator to be a specification of the required validations. public class PersonValidator extends GenericValidator { private static final List> VALIDATORS = new LinkedList<>(); static { VALIDATORS.add(person -> person.givenName != null); VALIDATORS.add(person -> person.familyName != null); VALIDATORS.add(person -> person.age != null); VALIDATORS.add(person -> person.gender != null); // ...and many more } public PersonValidator() { super(VALIDATORS); } } PersonValidator, and all your other validators, can now focus completely on validation. The behaviour hasn’t changed – the validation still fails fast. There’s no boiler plate, which is A Good Thing. This one’s going in the toolbox.

std::tuple is a very nice facility originally introduced in C++ TR1. It is a heterogenous container of elements that has a statically known size. In C++ 11, std::tuple can be implemented using variadic templates; a single std::tuple class can support an arbitrary number of template type arguments. In this series of blog posts we will implementstd::tuple from first principles. The purpose of this exercise is not to provide the best-performing or most-conformant tuple implementation, but rather to see what foundational concepts are required to implement it. NOTE: This blog series relies on good familiarity with variadic templates and basic template metaprogramming techniques. Consider yourself warned, or, even better, check out a gentle introduction to variadic templates and a great book with a thorough overview of template metaprogramming techniques. Before we start implementing std::tuple, let’s take a look at some of its functionality first. Here are a few things you can do with a tuple: tuple t1(42, 'a'); tuple t2(t1); get<0>(t1) = 43; get<1>(t2) = get<1>(t1) + 1; get(t2) = 43; // get added in C++ 14 set> tuples; // OK, tuple has operator< cout << tuple_size>::value; // prints '2' OK, so a tuple looks like any other container, except it has this weird syntax for accessing elements, with the get non-member function template. There are also some rather obvious operations like copy construction, move construction, assignment from another tuple, and so forth. There are also some slightly more complicated operations: tuple t3 = make_tuple("Hello", 42); tuple t4 = t3; // OK, heterogenous copy construction tuple t5 = tuple_cat(t3, t4); int x; string s; tie(x, s) = make_tuple(42, "hello"); // initialize both x and s tie(ignore, s) = make_tuple(-1, "goodbye"); // initialize only s So, let’s get started. The first challenge in implementing a tuple is figuring out its class declaration. What should it look like? It’s obviously a variadic class template, because you can create a tuple with an arbitrary number of types. Something like this, then? template class tuple; Looks good. The next question is: how do we represent the tuple’s elements? That is, what members should the tuple class have? Think about it. One tempting option is to have tuple derive from all its constituent types: template class tuple : Types... { }; Exercise 1: Why is that a bad idea? Well, it’s a bad idea for many reasons. Two obvious ones that come to mind: 1) you can’t derive from many types, like int and char and any sealed type; 2) you can’t derive from the same type multiple times, so you can’t have a tuple with this approach. What’s more, you’d end up polluting tuple’s interface with publicly accessible member functions from each of the constituent types. So, this is obviously not going to work. We are going to derive tuple of T1, …, Tn from n types, but they aren’t going to be T1, …, Tn themselves. Instead, we will need a wrapper type, which we’ll call tuple_element; tuple will derive from tuple_element n times. How should we declare tuple_element? How about this? template struct tuple_element { T value_; }; And then we can have tuple derive from tuple_elementn times, as follows: template class tuple : tuple_element... { }; Exercise 2: This is better than deriving from T1, …, Tn directly, but is still broken. Why? This is a nice attempt, but it still doesn’t work if some of the types are the same. Again, we if we instantiate a tuple, it can’t derive from tuple_element twice. Bummer. We obviously need some way to disambiguate the tuple_element base classes. How about adding an index? template struct tuple_element { T value_; }; Now, tuple_element<0, int> isn’t the same type as tuple_element<1, int>, so we can derive from both of them at once. The question is just how to do that. How can tuple derive from each of the tuple_elements with types T1, …, Tn and the appropriate indices 0, …, n-1? That is, what do we put in the following class definition? template class tuple : tuple_element... { }; This is an interesting problem. We need to replace the ??? placeholder with a parameter pack of size_ts. If we already had this kind of pack provided by the user, we could do the following: template class tuple : tuple_element... { }; Exercise 3: What if Indices and Types are parameter packs of different lengths? Answer: It’s a compilation error. However, if both parameter packs have the same length, the pack expansion operator (…) successfully expands each tuple_element base with an index and a type. Kind of like the functional ‘zip’ operator for two sequences. Clients can use this class as follows: tuple<0, 1, int, string> tup; Needless to say, it’s not very convenient. We want the client to provide only the types. We will have to generate the indices ourselves, somehow. Until next time.

If you have a requirement to save and serve files, then there are at least a couple options. Save the file onto the server and serve it from there. Mongo1 provide GridFS2 store that allows you not only to store files but also metadata related to the file. For example: you can store author, tags, group etc right with the file. You can provide this functionality via option 1 too, but you would need to make your own tables and link the files to the metadata information. Besides replication of data is in built in Mongo. Bottle You can upload and download mongo files using Bottle3 like so: import json from bottle import run, Bottle, request, response from gridfs import GridFS from pymongo import MongoClient FILE_API = Bottle() MONGO_CLIENT = MongoClient('mongodb://localhost:27017/') DB = MONGO_CLIENT['TestDB'] GRID_FS = GridFS(DB) @FILE_API.put('/upload/< file_name>') def upload(file_name): response.content_type = 'application/json' with GRID_FS.new_file(filename=file_name) as fp: fp.write(request.body) file_id = fp._id if GRID_FS.find_one(file_id) is not None: return json.dumps({'status': 'File saved successfully'}) else: response.status = 500 return json.dumps({'status': 'Error occurred while saving file.'}) @FILE_API.get('/download/< file_name>') def index(file_name): grid_fs_file = GRID_FS.find_one({'filename': file_name}) response.headers['Content-Type'] = 'application/octet-stream' response.headers["Content-Disposition"] = "attachment; filename={}".format(file_name) return grid_fs_file run(app=FILE_API, host='localhost', port=8080) And here's the break down of the code: Upload method: Line 12: Sets up upload method to recieve a PUT request for /upload/ url. Line 15-17: Create a new GridFS file with file_name and get the content from request.body. request.body may be StringIO type or a File type because Python is smart enough to decipher the body type based on the content. Line 18-19: If we can find the file by file name then it was saved successfully and therefore return a success response. Line 20-22: Return error if file was not saved successfully. Download method: Line 27: Find the GridFS file. Line 28-29: Set the response Content-Type as application-octet-stream and Content-Disposition to attachment; filename= Line 31: Return the GridOut object. Based on Bottle documentation below we can return an object which has .read() method available and Bottle understands that to be a File object. File objects Everything that has a .read() method is treated as a file or file-like object and passed to the wsgi.file_wrapper callable defined by the WSGI server framework. Some WSGI server implementations can make use of optimized system calls (sendfile) to transmit files more efficiently. In other cases this just iterates over chunks that fit into memory. And we are done (as far as Bottle is concerned). Flask You can upload/download files using Flask4 like so: import json from gridfs import GridFS from pymongo import MongoClient from flask import Flask, make_response from flask import request __author__ = 'ravihasija' app = Flask(__name__) mongo_client = MongoClient('mongodb://localhost:27017/') db = mongo_client['TestDB'] grid_fs = GridFS(db) @app.route('/upload/', methods=['PUT']) def upload(file_name): with grid_fs.new_file(filename=file_name) as fp: fp.write(request.data) file_id = fp._id if grid_fs.find_one(file_id) is not None: return json.dumps({'status': 'File saved successfully'}), 200 else: return json.dumps({'status': 'Error occurred while saving file.'}), 500 @app.route('/download/') def index(file_name): grid_fs_file = grid_fs.find_one({'filename': file_name}) response = make_response(grid_fs_file.read()) response.headers['Content-Type'] = 'application/octet-stream' response.headers["Content-Disposition"] = "attachment; filename={}".format(file_name) return response app.run(host="localhost", port=8081) The Flask upload and download code is very similar to Bottle. It differs only in a few places detailed below: Line 14: Routing is configured differently in Flask. You mention the URL and the methods that apply for that URL. Line 17: Instead of request.body you use request.data Line 28-31: Make the response with the file content and set up the appropriate headers. Finally, return the response object. Questions? Thoughts? Please feel free to leave me a comment below. Thank you for your time. Github repo: https://github.com/RaviH/file-upload-download-mongo References: MongoDB: http://www.mongodb.org/↩ GridFS: http://docs.mongodb.org/manual/core/gridfs/↩ Bottle: http://bottlepy.org/docs/dev/tutorial.html↩ Flask: http://flask.pocoo.org/↩ PyMongo GridFS doc http://api.mongodb.org/python/current/api/gridfs/index.html?highlight=gridfs#module-gridfs↩ Get to know GridFS: https://dzone.com/articles/get-know-gridfs-mongodb↩

Traditional web development builds a bridge between the front end, where the user performs their manipulations of the application’s data, and the back end, where that data is stored. In traditional web development, this process is driven by successive networking calls, communicating changes between the server and the client via re-rendering the involved pages. AngularJS enhances this with two-way data binding. Below we’ll look at what two-way data binding is, and how it differs from the traditional data processing approach. The Traditional Approach Most web frameworks focus on one-way data binding. This involves reading the input from the DOM, serializing the data, sending it to the server, waiting for the process to finish, then modifying the DOM to indicate any errors, or reloading the DOM if the call is successful. While this provides a traditional web application all the time it needs to perform data processing, this benefit is only really applicable to web apps with highly complex data structures. If your application has a simpler data format, with relatively flat models, then the extra work can needlessly complicate the process. Furthermore, all models need to wait for server confirmation before their data can be updated, meaning that related data depending upon those models won’t have the latest information. Tying Together the UI and the Model AngularJS addresses this with two-way data binding. With two-way data binding, the user interface changes are immediately reflected in the underlying data model, and vice-versa. This allows the data model to serve as an atomic unit that the view of the application can always depend upon to be accurate. Many web frameworks implement this type of data binding with a complex series of event listeners and event handlers – an approach that can quickly become fragile. AngularJS, on the other hand, makes this approach to data a primary part of its architecture. Instead of creating a series of callbacks to handle the changing data, AngularJS does this automatically without any needed intervention by the programmer Benefits and Considerations The primary benefit of two-way data binding is that updates to (and retrievals from) the underlying data store happen more or less automatically. When the data store updates, the UI updates as well. This allows you to remove a lot of logic from the front-end display code, particularly when making effective use of AngularJS’s declarative approach to UI presentation. In essence, it allows for true data encapsulation on the front-end, reducing the need to do complex and destructive manipulation of the DOM. While this solves a lot of problems with a website’s presentation architecture, there are some disadvantages to take into consideration. First, AngularJS uses a dirty-checking approach that can be slow in some browsers – not a problem for thin presentation pages, but any page with heavy processing may run into problems in older browsers. Additionally, two-way binding is only truly beneficial for relatively simple objects. Any data that requires heavy parsing work, or extensive manipulation and processing, will simply not work well with two-way binding. Additionally, some uses of Angular – such as using the same binding directive more than once – can break the data binding process. Conclusion While the traditional approach to data binding has a lot of benefits when it comes to performing complex data manipulations and calculations, it can introduce some problems with respect to the design of the web application’s front-end architecture. With AngularJS’s use of two-way data binding, your application can greatly simplify its presentation layer, allowing the UI to be built off of a cleaner, less-destructive approach to DOM presentation. While it isn’t useful in every situation, the two-way data binding AngularJS provides can greatly ease web application development, and reduce the pain faced by your front-end developers. Build your Angular app and connect it to any database with Backand today. – Get started now.

written by karthik appigatla for the mysql performance blog . a few years ago peter zaitsev, in a post titled “ to uuid or not to uuid ,” wrote: “ there is a timestamp based part in uuid which has similar properties to auto_increment and which could be used to have values generated at the same point in time physically local in btree index.” for this post i’ve rearranged the timestamp part of uuid (universal unique identifier) and did some benchmarks. many people store uuid as char (36) and use as row identity value (primary key) because it is unique across every table, every database and every server and allows easy merging of records from different databases. but here comes the problem, using it as a primary key causes the problems described below. problems with uuid uuid has 36 characters which makes it bulky. innodb stores data in the primary key order and all the secondary keys also contain primary key. so having uuid as primary key makes the index bigger which can not be fit into the memory inserts are random and the data is scattered. despite the problems with uuid, people still prefer it because it is unique across every table, can be generated anywhere. in this blog, i will explain how to store uuid in an efficient way by re-arranging timestamp part of uuid. structure of uuid mysql uses uuid version 1 which is a 128-bit number represented by a utf8 string of five hexadecimal numbers the first three numbers are generated from a timestamp. the fourth number preserves temporal uniqueness in case the timestamp value loses monotonicity (for example, due to daylight saving time). the fifth number is an ieee 802 node number that provides spatial uniqueness. a random number is substituted if the latter is not available (for example, because the host computer has no ethernet card, or we do not know how to find the hardware address of an interface on your operating system). in this case, spatial uniqueness cannot be guaranteed. nevertheless, a collision should have very low probability. the timestamp is mapped as follows: when the timestamp has the (60 bit) hexadecimal value: 1d8eebc58e0a7d7. the following parts of the uuid are set:: 58e0a7d7-eebc-11d8 -9669-0800200c9a66. the 1 before the most significant digits (in 11d8) of the timestamp indicates the uuid version, for time-based uuids this is 1. fourth and fifth parts would be mostly constant if it is generated from a single server. first three numbers are based on timestamp, so they will be monotonically increasing. lets rearrange the total sequence making the uuid closer to sequential. this makes the inserts and recent data look up faster. dashes (‘-‘) make no sense, so lets remove them. 58e0a7d7-eebc-11d8-9669-0800200c9a66 => 11d8eebc58e0a7d796690800200c9a66 benchmarking i created three tables: events_uuid – uuid binary(16) primary key events_int – additional bigint auto increment column and made it as primary key and index on uuid column events_uuid_ordered – rearranged uuid binary(16) as primary key i created three stored procedures which insert 25k random rows at a time into the respective tables. there are three more stored procedures which call the random insert-stored procedures in a loop and also calculate the time taken to insert 25k rows and data and index size after each loop. totally i have inserted 25m records. data size horizontal axis – number of inserts x 25,000 vertical axis – data size in mb the data size for uuid table is more than other two tables. index size horizontal axis – number of inserts x 25,000 vertical axis – index size in mb total size horizontal axis – number of inserts x 25,000 vertical axis – total size in mb time taken horizontal axis – number of inserts x 25,000 vertical axis – time taken in seconds for the table with uuid as primary key, you can notice that as the table grows big, the time taken to insert rows is increasing almost linearly. whereas for other tables, the time taken is almost constant. the size of uuid table is almost 50% bigger than ordered uuid table and 30% bigger than table with bigint as primary key. comparing the ordered uuid table bigint table, the time taken to insert rows and the size are almost same. but they may vary slightly based on the index structure. root@localhost:~# ls -lhtr /media/data/test/ | grep ibd -rw-rw---- 1 mysql mysql 13g jul 24 15:53 events_uuid_ordered.ibd -rw-rw---- 1 mysql mysql 20g jul 25 02:27 events_uuid.ibd -rw-rw---- 1 mysql mysql 15g jul 25 07:59 events_int.ibd table structure #1 events_int create table `events_int` ( `count` bigint(20) not null auto_increment, `id` binary(16) not null, `unit_id` binary(16) default null, `event` int(11) default null, `ref_url` varchar(255) collate utf8_unicode_ci default null, `campaign_id` binary(16) collate utf8_unicode_ci default '', `unique_id` binary(16) collate utf8_unicode_ci default null, `user_agent` varchar(100) collate utf8_unicode_ci default null, `city` varchar(80) collate utf8_unicode_ci default null, `country` varchar(80) collate utf8_unicode_ci default null, `demand_partner_id` binary(16) default null, `publisher_id` binary(16) default null, `site_id` binary(16) default null, `page_id` binary(16) default null, `action_at` datetime default null, `impression` smallint(6) default null, `click` smallint(6) default null, `sold_impression` smallint(6) default null, `price` decimal(15,7) default '0.0000000', `actioned_at` timestamp not null default '0000-00-00 00:00:00', `unique_ads` varchar(255) collate utf8_unicode_ci default null, `notification_url` text collate utf8_unicode_ci, primary key (`count`), key `id` (`id`), key `index_events_on_actioned_at` (`actioned_at`), key `index_events_unit_demand_partner` (`unit_id`,`demand_partner_id`) ) engine=innodb default charset=utf8 collate=utf8_unicode_ci; #2 events_uuid create table `events_uuid` ( `id` binary(16) not null, `unit_id` binary(16) default null, ~ ~ primary key (`id`), key `index_events_on_actioned_at` (`actioned_at`), key `index_events_unit_demand_partner` (`unit_id`,`demand_partner_id`) ) engine=innodb default charset=utf8 collate=utf8_unicode_ci; #3 events_uuid_ordered create table `events_uuid_ordered` ( `id` binary(16) not null, `unit_id` binary(16) default null, ~ ~ primary key (`id`), key `index_events_on_actioned_at` (`actioned_at`), key `index_events_unit_demand_partner` (`unit_id`,`demand_partner_id`) ) engine=innodb default charset=utf8 collate=utf8_unicode_ci; conclusions create function to rearrange uuid fields and use it delimiter // create definer=`root`@`localhost` function `ordered_uuid`(uuid binary(36)) returns binary(16) deterministic return unhex(concat(substr(uuid, 15, 4),substr(uuid, 10, 4),substr(uuid, 1, 8),substr(uuid, 20, 4),substr(uuid, 25))); // delimiter ; inserts insert into events_uuid_ordered values (ordered_uuid(uuid()),'1','m',....); selects select hex(uuid),is_active,... from events_uuid_ordered ; define uuid as binary(16) as binary does not have any character set references http://dev.mysql.com/doc/refman/5.1/en/miscellaneous-functions.html#function_uuid http://www.famkruithof.net/guid-uuid-timebased.html http://www.percona.com/blog/2007/03/13/to-uuid-or-not-to-uuid/ http://blog.codinghorror.com/primary-keys-ids-versus-guids/

My objective here is to recreate a similar set-up in a smaller unit test mode.