Within Spring Integration, one form of EIP scatter-gather is provided by the splitter and aggregator constructs.

in working with development teams at organizations which are adopting continuous delivery , i have found there can be friction over practices that many developers have come to consider as the right way for agile teams to work. i believe the root of conflicts between what i’ve come to think of as traditional agile and cd is the approach to making software “ready for release”. evolution of software delivery a usefully simplistic view of the evolution of ideas about making software ready for release is this: waterfall believes a team should only start making its software ready for release when all of the functionality for the release has been developed (i.e. when it is “feature complete”). agile introduces the idea that the team should get their software ready for release throughout development. many variations of agile (which i refer to as “traditional agile” in this post) believe this should be done at periodic intervals. continuous delivery is another subset of agile which in which the team keeps its software ready for release at all times during development. it is different from “traditional” agile in that it does not involve stopping and making a special effort to create a releasable build. continuous delivery is not about shorter cycles going from traditional agile development to continuous delivery is not about adopting a shorter cycle for making the software ready for release. making releasable builds every night is still not continuous delivery. cd is about moving away from making the software ready as a separate activity, and instead developing in a way that means the software is always ready for release. ready for release does not mean actually releasing a common misunderstanding is that continuous delivery means releasing into production very frequently. this confusion is made worse by the use of organizations that release software multiple times every day as poster children for cd. continuous delivery doesn’t require frequent releases, it only requires ensuring software could be released with very little effort at any point during development. (see jez humble’s article on continuous delivery vs. continuous deployment .) although developing this capability opens opportunities which may encourage the organization to release more often, many teams find more than enough benefit from cd practices to justify using it even when releases are fairly infrequent. friction points between continuous delivery and traditional agile as i mentioned, there are sometimes conflicts between continuous delivery and practices that development teams take for granted as being “proper” agile. friction point: software with unfinished work can still be releasable one of these points of friction is the requirement that the codebase not include incomplete stories or bugfixes at the end of the iteration. i explored this in my previous post on iterations . this requirement comes from the idea that the end of the iteration is the point where the team stops and does the extra work needed to prepare the software for release. but when a team adopts continuous delivery, there is no additional work needed to make the software releasable. more to the point, the cd team ensures that their code could be released to production even when they have work in progress, using techniques such as feature toggles . this in turn means that the team can meet the requirement that they be ready for release at the end of the iteration even with unfinished stories. this can be a bit difficult for people to swallow. the team can certainly still require all work to be complete at the iteration boundary, but this starts to feel like an arbitrary constraint that breaks the team’s flow. continuous delivery doesn’t require non-timeboxed iterations, but the two practices are complementary. friction point: snapshot/release builds many development teams divide software builds into two types, “snapshot” builds and “release” builds. this is not specific to agile, but has become strongly embedded in the java world due to the rise of maven, which puts the snapshot/build concept at the core of its design. this approach divides the development cycle into two phases, with snapshots being used while software is in development, and a release build being created only when the software is deemed ready for release. this division of the release cycle clearly conflicts with the continuous delivery philosophy that software should always be ready for release. the way cd is typically implemented involves only creating a build once, and then promoting it through multiple stages of a pipeline for testing and validation activities, which doesn’t work if software is built in two different ways as with maven. it’s entirely possible to use maven with continuous delivery, for example by creating a release build for every build in the pipeline. however this leads to friction with maven tools and infrastructure that assume release builds are infrequent and intended for production deployment. for example, artefact repositories such as nexus and artefactory have housekeeping features to delete old snapshot builds, but don’t allow release builds to be deleted. so an active cd team, which may produce dozens of builds a day, can easily chew through gigabytes and terabytes of disk space on the repository. friction point: heavier focus on testing deployability a standard practice with continuous delivery is automatically deploying every build that passes basic continuous integration to an environment that emulates production as closely as possible, using the same deployment process and tooling. this is essential to proving whether the code is ready for release on every commit, but this is more rigorous than many development teams are used to having in their ci. for example, pre-cd continuous integration might run automated functional tests against the application by deploying it to an embedded application server using a build tool like ant or maven. this is easier for developers to use and maintain, but is probably not how the application will be deployed in production. so a cd team will typically add an automated deployment to an environment will more fully replicates production, including separated web/app/data tiers, and deployment tooling that will be used in production. however this more production-like deployment stage is more likely to fail due to its added complexity, and may be may be more difficult for developers to maintain and fix since it uses tooling more familiar to system administrators than to developers. this can be an opportunity to work more closely with the operations team to create a more reliable, easily supported deployment process. but it is likely to be a steep curve to implement and stabilize this process, which may impact development productivity. is cd worth it? given these friction points, what benefit is there to moving from traditional agile to continuous delivery worthwhile, especially for a team that is unlikely to actually release into production more often than every iteration? decrease risk by uncovering deployment issues earlier, increase flexibility by giving the organization the option to release at any point with minimal added cost or risk, involves everyone involved in production releases - such as qa, operations, etc. - in making the full process more efficient. the entire organization must identify difficult areas of the process and find ways to fix them, through automation, better collaboration, and improved working practices, by continuously rehearsing the release process, the organization becomes more competent at doing it, so that releasing becomes autonomic, like breathing, rather than traumatic, like giving birth, improves the quality of the software, by forcing the team to fix problems as they are found rather than being able to leave things for later. dealing with the friction the friction points i’ve described seem to come up fairly often when continuous delivery is being introduced. my hope is that understanding the source of this friction will be helpful in discussing it when it comes up, and working through the issues. if developers who are initially uncomfortable with breaking with the “proper” way of doing things, or find a cd pipeline overly complex or difficult understand the aims and value of these practices, hopefully they will be more open to giving them a chance. once these practices become embedded and mature in an organization, team members often find it’s difficult to go back to the old ways of doing them. edit: i’ve rephrased the definition of the “traditional agile” approach to making software ready for release. this definition is not meant to apply to all agile practices, but rather applies to what seems to me to be a fairly mainstream belief that agile means stopping work to make the software releasable.

Insanity: doing the same thing over and over again and expecting different results. -- attributed to Albert Einstein Global state is essentially the same thing as insanity in this definition: a way to affect the execution of code hidden from sight, so that two apparently identical lines actually produce a different result depending on some external factor. For example: new SomeClass()->printId(); new SomeClass()->printId(); // output: 1, 2 has some global state (a static counter) affecting a field inside SomeClass instances. Therefore, it may not be easy to replicate scenarios (like in tests) multiple times. Examples Global and environmental variables, along with constants are simple examples of global state. The same goes for configuration directives and files which code silently depends upon, as long as they are global for each instance of the affected objects. Speaking about objects singletons and static classes containing fields are another example of global state. More subtle cases are hidden localizations like translations of output and of symbols (LC_ALL?) Parameterization is made difficult by global state, either because the seam for collaborators is hidden (config files and enviromental/global variables) or not accessible (singletons). Testability When there is some global state in an application, the affected unit tests won'tbe isolated from each other, and may change their results when run alone or in a different order with respect to being executed inside the whole test suite. Global state is one of the most common problems while working with legacy code which was written with little concern for testability (and separation of concerns). A typical annoying example is a test that passes when alone, but fails in the full suite due to some state left lingering from previous tests. Usually, it is then debugged by executing the exact same test twice or multiple times in a single process and verifying that it passes consistently. Actually global state cannot be always removed, even in a test environment: what this move would achieve would be a fully parameterized system, too general to be useful; imagine configuring every class name in your application, even in Factories. It may be simpler to test with some global state in, like in the case of a default locale defined in place of stubbing the Translator object; or in the case of a Fake database connection instead of a Stub or a Mock. Taking this approach to the extreme, we notice that global state is often hidden from our view because it's taken for granted. Base classes offered by the language (e.g. String) are not mocked or substituted by test doubles, even when they have quite some logic in them; all the classes and functions contained in our applications are global state as their implementation cannot be substituted, yet we don't consider them a trouble as singletons. Constant There is a reason why not all global state is necessarily bad: constant global state to allow context-free reasoning about code, and simplify testing and reuse of code considerably. In fact, the very definition of state (for example from hardware logic networks) is that of a component that can change its behavior in time, keeping information about previous inputs. In short, any computation that is frequently accessed but does not have the capability to change its result or to produce side-effects is not state (it is global). ROM is instead considered a purely combinatorial network, not being real "memory" but merely a function translating addresses to words. A singleton changing its responses after some calls is global state that makes testing difficult; a static class containing only pure functions may make tests long winded and infringe the object-oriented paradigm, but it's not as dangerous as the former. However, that's why I see monkey patching in dynamic languages as problematic. Monkey patching commonly consists of open classes where you can add methods at any time after their initial definition. class Array def sum inject {|sum, x| sum + x } end end When you see a call to this method, you have to ask some questions: where it was added in the code base? Which sourcefile should I look at? When it was added to the code base? Am I sure that definitions can only be included at startup and I am not calling the method before it is defined? Are there multiple redefinitions of the method? Maybe from other libraries or code to integrate? The same issues happened for prototype.js, which modifies the prototype object of base JavaScript objects like Array, effectively redefining and adding methods. The result is little interoperability with other libraries. But even monkey patching should be fine as long as the modifications are really constant and definitive. If you use a single framework (like Ext Js), and you always use it in all pages of the application and in tests, then it can monkeypatch the base classes of the language safely, without making you debug a method that works in one environment but not in the other. Conclusion Global state is not only a global parameter for the internals of your application, but also the product of stateful interaction that changes the output and side-effects of code in different invocations. Making global things constant is the first step towards simplifying reasoning about a design and raising testability. If you are able to run a unit test twice in the same method, you are officially free from global state in that scenario. Pay attention when embracing open classes, editable prototypes, embedded calls to registries and files, and so on: they can add several dimensions to the variables that can affect the result of a piece of code. They will hide a dependency but not making it go away.

Learn how Apache Camel implements the EIPs and offers a standardized, internal domain-specific language (DSL) to integrate applications.

An Alpha Animation is animation that controls the alpha level of an object, i.e. fading it in and out. In Android, you can apply that fading effect to almost anything, from simple text, to images, buttons, check boxes, etc... Android has a few classes that can help you add that special effect to your programs, like AlphaAnimation and AnimationUtils. Here's an example on how to apply fading on any Android component subclass of View. First, the XML resource. In the resources folder, we will create a tiny XML configuration file with the characteristics of the fading effect we want in an "anim" subfolder. So, under res/anim, here's our alpha.xml: We are choosing to have a very basic full fade in effect (alpha from 0 to 1) that lasts one second. The above can also be done directly in Java code: Animation animation = new AlphaAnimation(0.0f, 1.0f); animation.setDuration(1000); Configuring the animation in resources or in code is ultimately a matter of preference. We will use the XML in this example. This is our class that does the above fading to any View (TextView, Button, etc..): package com.ts.fx.utils; import android.app.Activity; import android.view.View; import android.view.animation.Animation; import android.view.animation.AnimationUtils; public class Fader { /** * handles all subclasses of View : TextView, Button, ImageView etc.. * given the component's id in their layout file * */ public static void runAlphaAnimation(Activity act, int viewId) { // load animation XML resource under res/anim Animation animation = AnimationUtils.loadAnimation(act, R.anim.alpha); if(animation == null){ return; // here, we don't care } // reset initialization state animation.reset(); // find View by its id attribute in the XML View v = act.findViewById(viewId); // cancel any pending animation and start this one if (v != null){ v.clearAnimation(); v.startAnimation(animation); } } } The runAlphaAnimation() method takes an Activity reference and a View id attribute (as set up in the View's layout XML). We're basically done. all we have to do now is call it from any one of our Activitites: // inside an Activity with text, checkbox and button Fader.runAlphaAnimation(this, a_text.getId()); Fader.runAlphaAnimation(this, a_checkbox.getId()); Fader.runAlphaAnimation(this, a_button.getId()); //etc... That's all there is to it. The same basic technique seen here applies to all other special effects like translating, scaling or rotating components. The Animation classes have of course lots of other cool stuff, like controlling the z-ordering of the animated components, acceleration and repeat effects. Here's a thirty-second video (by yours truly) demonstrating various Android special effects (fading, translation and rotation) used together in a concrete application: From Tony's Blog.

Continuing on the theme of temporary storage for transient messages used within Spring Integration flows, the claim-check model offers configurable storage for message payloads. The advantage in using this Enterprise Integration pattern, compared against header enrichment, is that objects don't have to be packed into the header using a Header Enrichment technique. They can be stored in a local Java Map, an IMDB, cache or anything else that be used to hold data. Several advantages using this approach are evident. Firstly, performance and efficiency. When using header enrichment, if message payloads need to be managed outside of the JVM that generates the enriched message header, the object will not be available unless it's serialised and transported around the distributed application. This could be costly in terms of performance and transport efficiency. The key factor here is the frequency of remote dispatch and the size of the header object. In specific circumstances the claim-check pattern may offer an advantage here, objects can be serialised and/or transformed into a storage specific format and stored internally in memory or externally in a data store. Secondly, accessibility. It's conceivable that message payloads undergoing claim-check processing may need to be accessed by third party applications that are unable to receive Spring Integration messages. The claim-check pattern allows this type of processing to take place. Thirdly, resiliency is offered. A data store can be chosen that guarantees persistence for messages in order that they can be recovered following failure. The following code details how the claim-check pattern can be used: The gateway used is specified as the following Java class: package com.l8mdv.sample; import org.springframework.integration.Message; import org.springframework.integration.annotation.Gateway; public interface ClaimCheckGateway { public static final String CLAIM_CHECK_ID = "CLAIM_CHECK_ID"; @Gateway (requestChannel = "claim-check-in-channel") public Message send(Message message); } Lastly, this can all be tested by using the following JUnit test case: package com.l8mdv.sample; import org.junit.Assert; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.integration.Message; import org.springframework.integration.support.MessageBuilder; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import static com.l8mdv.sample.ClaimCheckGateway.CLAIM_CHECK_ID; @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration( locations = {"classpath:META-INF/spring/claim-check.xml"} ) public class ClaimCheckIntegrationTest { @Autowired ClaimCheckGateway claimCheckGateway; @Test public void locatePayloadInHeader() { String payload = "Sample test message."; Message message = MessageBuilder.withPayload(payload).build(); Message response = claimCheckGateway.send(message); Assert.assertTrue(response.getPayload().equals(payload)); Assert.assertTrue(response.getHeaders().get(CLAIM_CHECK_ID) != null); } }

A few weeks ago, I co-authored an article (with my colleague Rama Ramani) about how the Screen Actors Guild Awards website migrated its Drupal deployment from LAMP to Windows Azure: Azure Real World: Migrating a Drupal Site from LAMP to Windows Azure. Since then, Rama and another colleague, Jason Roth, have been working on writing up how the SAG Awards website was managed and monitored in Windows Azure. The article below is the fruit of their work…a very interesting/educational read. Overview Drupal is an open source content management system that runs on PHP. Windows Azure offers a flexible platform for hosting, managing, and scaling Drupal deployments. This paper focuses on an approach to host Drupal sites on Windows Azure, based on learning from a BPD Customer Programs Design Win engagement with the Screen Actors Guild Awards Drupal website. This paper covers guidelines and best practices for managing an existing Drupal web site in Windows Azure. For more information on how to migrate Drupal applications to Windows Azure, see Azure Real World: Migrating a Drupal Site from LAMP to Windows Azure. The target audience for this paper is Drupal administrators who have some exposure to Windows Azure. More detailed pointers to Windows Azure content is provided throughout the paper as links. Drupal Application Architecture on Windows Azure Before reviewing the management and monitoring guidelines, it is important to understand the architecture of a typical Drupal deployment on Windows Azure. First, the following diagram displays the basic architecture of Drupal running on Windows and IIS7. In the Windows Server scenario, you could have one or more machines hosting the web site in a farm. Those machines would either persist the site content to the file system or point to other network shares. For Windows Azure, the basic architecture is the same, but there are some differences. In Windows Azure the site is hosted on a web role. A web role instance is hosted on a Windows Server 2008 virtual machine within the Windows Azure datacenter. Like the web farm, you can have multiple instances running the site. But there is no persistence guarantee for the data on the file system. Because of this, much of the shared site content should be stored in Windows Azure Blob storage. This allows them to be highly available and durable. Usually, a large portion of the site caters to static content which lends well to caching. And caching can be applied in a set of places – browser level caching, CDN to cache content in the edge closer to the browser clients, caching in Azure to reduce the load on backend, etc. Finally, the database can be located in SQL Azure. The following diagram shows these differences. For monitoring and management, we will look at Drupal on Windows Azure from three perspectives: Availability: Ensure the web site does not go down and that all tiers are setup correctly. Apply best practices to ensure that the site is deployed across data centers and perform backup operations regularly. Scalability: Correctly handle changes in user load. Understand the performance characteristics of the site. Manageability: Correctly handle updates. Make code and site changes with no downtime when possible. Although some management tasks span one or more of these categories, it is still helpful to discuss Drupal management on Windows Azure within these focus areas. Availability One main goal is that the Drupal site remains running and accessible to all end-users. This involves monitoring both the site and the SQL Azure database that the site depends on. In this section, we will briefly look at monitoring and backup tasks. Other crossover areas that affect availability will be discussed in the next section on scalability. Monitoring With any application, monitoring plays an important role with managing availability. Monitoring data can reveal whether users are successfully using the site or whether computing resources are meeting the demand. Other data reveals error counts and possibly points to issues in a specific tier of the deployment. There are several monitoring tools that can be used. The Windows Azure Management Portal. Windows Azure diagnostic data. Custom monitoring scripts. System Center Operations Manager. Third party tools such as Azure Diagnostics Manager and Azure Storage Explorer. The Windows Azure Management Portal can be used to ensure that your deployments are successful and running. You can also use the portal to manage features such as Remote Desktop so that you can directly connect to machines that are running the Drupal site. Windows Azure diagnostics allows you to collect performance counters and logs off of the web role instances that are running the Drupal site. Although there are many options for configuring diagnostics in Azure, the best solution with Drupal is to use a diagnostics configuration file. The following configuration file demonstrates some basic performance counters that can monitor resources such as memory, processor utilization, and network bandwidth. For more information about setting up diagnostic configuration files, see How to Use the Windows Azure Diagnostics Configuration File. This information is stored locally on each role instance and then transferred to Windows Azure storage per a defined schedule or on-demand. See Getting Started with Storing and Viewing Diagnostic Data in Windows Azure Storage. Various monitoring tools, such as Azure Diagnostics Manager, help you to more easily analyze diagnostic data. Monitoring the performance of the machines hosting the Drupal site is only part of the story. In order to plan properly for both availability and scalability, you should also monitor site traffic, including user load patterns and trends. Standard and custom diagnostic data could contribute to this, but there are also third-party tools that monitor web traffic. For example, if you know that spikes occur in your application during certain days of the week, you could make changes to the application to handle the additional load and increase the availability of the Drupal solution. Backup Tasks To remain highly available, it is important to backup your data as a defense-in-depth strategy for disaster recovery. This is true even though SQL Azure and Windows Azure Storage both implement redundancy to prevent data loss. One obvious reason is that these services cannot prevent administrator error if data is accidentally deleted or incorrectly changed. SQL Azure does not currently have a formal backup technology, although there are many third-party tools and solutions that provide this capability. Usually the database size for a Drupal site is relatively small. In the case of SAG Awards, it was only ~100-150 MB. So performing an entire backup using any strategy was relatively fast. If your database is much larger, you might have to test various backup strategies to find the one that works best. Apart from third-party SQL Azure backup solutions, there are several strategies for obtaining a backup of your data: · Use the Drush tool and the portabledb-export command. · Periodically copy the database using the CREATE DATABASE Transact-SQL command. · Use Data-tier applications (DAC) to assist with backup and restore of the database. SQL Azure backup and data security techniques are described in more detail in the topic, Business Continuity in SQL Azure. Note that bandwidth costs accrue with any backup operation that transfers information outside of the Windows Azure datacenter. To reduce costs, you can copy the database to a database within the same datacenter. Or you can export the data-tier applications to blob storage in the same datacenter. Another potential backup task involves the files in Blob storage. If you keep a master copy of all media files uploaded to Blob storage, then you already have an on-premises backup of those files. However, if multiple administrators are loading files into Blob storage for use on the Drupal site, it is a good idea to enumerate the storage account and to download any new files to a central location. The following PHP script demonstrates how this can be done by backing up all files in Blob storage after a specified modification date. setProxy(true, 'YOUR_PROXY_IF_NEEDED', 80); $blobs = (array)$blobObj->listBlobs(AZURE_STORAGE_CONTAINER, '', '', 35000); backupBlobs($blobs, $blobObj); function backupBlobs($blobs, $blobObj) { foreach ($blobs as $blob) { if (strtotime($blob->lastmodified) >= DEFAULT_BACKUP_FROM_DATE && strtotime($blob->lastmodified) <= DEFAULT_BACKUP_TO_DATE) { $path = pathinfo($blob->name); if ($path['basename'] != '$$$.$$$') { $dir = $path['dirname']; $oldDir = getcwd(); if (handleDirectory($dir)) { chdir($dir); $blobObj->getBlob( AZURE_STORAGE_CONTAINER, $blob->name, $path['basename'] ); chdir($oldDir); } } } } } function handleDirectory($dir) { if (!checkDirExists($dir)) { return mkdir($dir, 0755, true); } return true; } function checkDirExists($dir) { if(file_exists($dir) && is_dir($dir)) { return true; } return false; } ?> This script has a dependency on the Windows Azure SDK for PHP. Also note there are several parameters that you must modify such as the storage account, secret, and backup location. As with SQL Azure, bandwidth and transaction charges apply to a backup script like this. Scalability Drupal sites on Windows Azure can scale as load increased through typical strategies of scale-up, scale-out, and caching. The following sections describe the specifics of how these strategies are implemented in Windows Azure. Typically you make scalability decisions based on monitoring and capacity planning. Monitoring can be done in staging during testing or in production with real-time load. Capacity planning factors in projections for changes in user demand. Scale Up When you configure your web role prior to deployment, you have the option of specifying the Virtual Machine (VM) size, such as Small or ExtraLarge. Each size tier adds additional memory, processing power, and network bandwidth to each instance of your web role. For cost efficiency and smaller units of scale, you can test your application under expected load to find the smallest virtual machine size that meets your requirements. The workload usually in most popular Drupal websites can be separated out into a limited set of Drupal admins making content changes and a large user base who perform mostly read-only workload. End users can be allowed to make ‘writes’, such as uploading blogs or posting in forums, but those changes are not ‘content changes’. Drupal admins are setup to operate without caching so that the writes are made directly to SQL Azure or the corresponding backend database. This workload performs well with Large or ExtraLarge VM sizes. Also, note that the VM size is closely tied to all hardware resources, so if there are many content-rich pages that are streaming content, then the VM size requirements are higher. To make changes to the Virtual Machine size setting, you must change the vmsize attribute of the WebRole element in the service definition file, ServiceDefinition.csdef. A virtual machine size change requires existing applications to be redeployed. Scale Out In addition to the size of each web role instance, you can increase or decrease the number of instances that are running the Drupal site. This spreads the web requests across more servers, enabling the site to handle more users. To change the number of running instances of your web role, see How to Scale Applications by Increasing or Decreasing the Number of Role Instances. Note that some configuration changes can cause your existing web role instances to recycle. You can choose to handle this situation by applying the configuration change and continue running. This is done by handling the RoleEnvironment.Changing event. For more information see, How to Use the RoleEnvironment.Changing Event. A common question for any Windows Azure solution is whether there is some type of built-in automatic scaling. Windows Azure does not provide a service that provides auto-scaling. However, it is possible to create a custom solution that scales Azure services using the Service Management API. For an example of this approach, see An Auto-Scaling Module for PHP Applications in Windows Azure. Caching Caching is an important strategy for scaling Drupal applications on Windows Azure. One reason for this is that SQL Azure implements throttling mechanisms to regulate the load on any one database in the cloud. Code that uses SQL Azure should have robust error handling and retry logic to account for this. For more information, see Error Messages (SQL Azure Database). Because of the potential for load-related throttling as well as for general performance improvement, it is strongly recommended to use caching. Although Windows Azure provides a Caching service, this service does not currently have interoperability with PHP. Because of this, the best solution for caching in Drupal is to use a module that uses an open-source caching technology, such as Memcached. Outside of a specific Drupal module, you can also configure Memcached to work in PHP for Windows Azure. For more information, see Running Memcached on Windows Azure for PHP. Here is also an example of how to get Memcached working in Windows Azure using a plugin: Windows Azure Memcached plugin. In a future paper, we hope to cover this architecture in more detail. For now, here are several design and management considerations related to caching. Area Consideration Design and Implementation For a technology like Memcached, will the cache be collocated (spread across all web role instances)? Or will you attempt to setup a dedicated cache ring with worker roles that only run Memcached? Configuration What memory is required and how will items in the cache be invalidated? Performance and Monitoring What mechanisms will be used to detect the performance and overall health of the cache? For ease of use and cost savings, collocation of the cache across the web role instances of the Drupal site works best. However, this assumes that there is available reserve memory on each instance to apply toward caching. It is possible to increase the virtual machine size setting to increase the amount of available memory on each machine. It is also possible to add additional web role instances to add to the overall memory of the cache while at the same time improving the ability of the web site to respond to load. It is possible to create a dedicated cache cluster in the cloud, but the steps for this are beyond the scope of this paper[RR1] . For Windows Azure Blob storage, there is also a caching feature built into the service called the Content Delivery Network (CDN). CDN provides high-bandwidth access to files in Blob storage by caching copies of the files in edge nodes around the world. Even within a single geographic region, you could see performance improvements as there are many more edge nodes than Windows Azure datacenters. For more information, see Delivering High-Bandwidth Content with the Windows Azure CDN. Manageability It is important to note that each hosted service has a Staging environment and a Production environment. This can be used to manage deployments, because you can load and test and application in staging before performing a VIP swap with production. From a manageability standpoint, Drupal has an advantage on Windows Azure in the way that site content is stored. Because the data necessary to serve pages is stored in the database and blob storage, there is no need to redeploy the application to change the content of the site. Another best practice is to use a separate storage account for diagnostic data than the one that is used for the application itself. This can improve performance and also helps to separate the cost of diagnostic monitoring from the cost of the running application. As mentioned previously, there are several tools that can assist with managing Windows Azure applications. The following table summarizes a few of these choices. Tool Description Windows Azure Management Portal The web interface of the Windows Azure management portal shows deployments, instance counts and properties, and supports many different common management and monitoring tasks. Azure Diagnostics Managerq[RR2] [JR3] A Red Gate Software product that provides advanced monitoring and management of diagnostic data. This tool can be very useful for easily analyzing the performance of the Drupal site to determine appropriate scaling decisions. Azure Storage Explorer A tool created by Neudesic for viewing Windows Azure storage account. This can be useful for viewing both diagnostic data and the files in Blob storage.

An old customer recently asked me if I had a solution for how to integrate between their existing JMS infrastructure on Websphere MQ with RabbitMQ. Although I know that RabbitMQ has the shovel plugin which can bridge between Rabbit instances I've yet not found a good plugin for JMS <-> AMQP forwarding. The first thing that came to my mind was to utilize a Spring Integration mediation as SI has excellent support for both JMS and Rabbit. Curious as I am I started a PoC and this is the result. It takes messages of a JMS queue and forwards to an AMQP exchange that is bound to a queue the consumer application is supposed to listen to. I used an external HornetQ instance in JBoss 6.1 as the JMS Provider, but I am 100% secure that the same setup would work for Websphere MQ as they both implement JMS. Be aware that I've done no performance tweaking or QoS setup yet as this is just a proof-of-concept. For a real setup you'd probably have to think about delivery guarantees versus performance and etc... The code will be available at a GitHub repository near you soon.. SpringContext in XML: org.jnp.interfaces.NamingContextFactory jnp://localhost:1099 org.jnp.interfaces:org.jboss.naming ConnectionFactory Maven POM: 4.0.0 org.rl si.jmstorabbit 0.0.1-SNAPSHOT jar si.jmstorabbit http://maven.apache.org UTF-8 2.2.5.Final 2.1.0.RELEASE springsource-release http://repository.springsource.com/maven/bundles/release false springsource-external http://repository.springsource.com/maven/bundles/external false org.springframework.integration spring-integration-core ${spring.integration.version} org.springframework.integration spring-integration-file ${spring.integration.version} org.springframework.integration spring-integration-amqp ${spring.integration.version} org.springframework.integration spring-integration-jms ${spring.integration.version} junit junit 3.8.1 test org.springframework spring-context 3.0.7.RELEASE jboss jnp-client 4.2.2.GA org.hornetq hornetq-core-client ${hornet.version} org.hornetq hornetq-jms-client ${hornet.version} org.hornetq hornetq-jms ${hornet.version} jboss jboss-common-client 3.2.3 org.jboss.netty netty 3.2.7.Final javax.jms jms 1.1

See original post at https://muhammadkhojaye.blogspot.com/2012/04/how-to-run-amazon-elastic-mapreduce-job.html Introduction Amazon EMR is a web service which can be used to easily and efficiently process enormous amounts of data. It uses a hosted Hadoop framework running on the web-scale infrastructure of Amazon EC2 and Amazon S3. Amazon EMR removes most of the cumbersome details of Hadoop while taking care of provisioning of Hadoop, running the job flow, terminating the job flow, moving the data between Amazon EC2 and Amazon S3, and optimizing Hadoop. In this tutorial, we will use a developed WordCount Java example using Hadoop and thereafter, we execute our program on Amazon Elastic MapReduce. Prerequisites You must have valid AWS account credentials. You should also have a general familiarity with using the Eclipse IDE before you begin. The reader can also use any other IDE of their choice. Step 1 – Develop MapReduce WordCount Java Program In this section, we are first going to develop a WordCount application. A WordCount program will determine how many times different words appear in a set of files. In Eclipse (or whatever the IDE you are using), Create simple Java Project with the name "WordCount". Create a java class name Map and override the map method as follow, public class Map extends Mapper { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); @Override public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } } Create a java class named Reduce and override the reduce method as shown below, public class Reduce extends Reducer { @Override protected void reduce(Text key, java.lang.Iterable values, org.apache.hadoop.mapreduce.Reducer.Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable value : values) { sum += value.get(); } context.write(key, new IntWritable(sum)); } } Create a java class named WordCount and defined the main method as below, public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); job.setJarByClass(WordCount.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitForCompletion(true); } Export the WordCount program in a jar using eclipse and save it to some location on disk. Make sure that you have provided the Main Class (WordCount.jar) during extraction ofu8u the jar file as shown below. Our jar is ready!!! Step 2 – Upload the WordCount JAR and Input Files to Amazon S3 Now we are going to upload the WordCount jar to Amazon S3. First, go to the following URL: https://console.aws.amazon.com/s3/home Next, click “Create Bucket”, give your bucket a name, and click the “Create” button. Select your new S3 bucket in the left-hand pane. Upload the WordCount JAR and sample input file for counting the words. Step 3 – Running an Elastic MapReduce job Now that the JAR is uploaded into S3, all we need to do is to create a new Job flow. let's execute the steps below. (I encourage readers to check out the following link for details regarding each step, How to Create a Job Flow Using a Custom JAR ) Sign in to the AWS Management Console and open the Amazon Elastic MapReduce console at https://console.aws.amazon.com/elasticmapreduce/ Click Create New Job Flow. In the DEFINE JOB FLOW page, enter the following details, a) Job Flow Name = WordCountJob b) Select Run your own applications) Select Custom JAR in the drop-down list) Click Continue In the SPECIFY PARAMETERS page, enter values in the boxes using the following table as a guide, and then click Continue.JAR Location = bucketName/jarFileLocationJAR Arguments =s3n://bucketName/inputFileLocations3n://bucketName/outputpath Please note that the output path must be unique each time we execute the job. The Hadoop always create a folder with the same name specified here. After executing the job, just wait and monitor your job that runs through the Hadoop flow. You can also look for errors by using the Debug button. The job should be complete within 10 to 15 minutes (can also depend on the size of the input). After completing the job, You can view results in the S3 Browser panel. You can also download the files from S3 and can analyze the outcome of the job. Amazon Elastic MapReduce Resources Amazon Elastic MapReduce Documentation,http://aws.amazon.com/documentation/elasticmapreduce/ Amazon Elastic MapReduce Getting Started Guide,http://docs.amazonwebservices.com/ElasticMapReduce/latest/GettingStartedGuide/ Amazon Elastic MapReduce Developer Guide,http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/ Apache Hadoop,http://hadoop.apache.org/ See more at https://muhammadkhojaye.blogspot.com/2012/04/how-to-run-amazon-elastic-mapreduce-job.html

Let's take a closer look at how to play sounds on an Android device with SoundPool and MediaPlayer.

Sometimes Quartz is not capable of running your job at the time when you desired. There are three reasons for that: all worker threads were busy running other jobs (probably with higher priority) the scheduler itself was down the job was scheduled with start time in the past (probably a coding error) You can increase the number of worker threads by simply customizing the org.quartz.threadPool.threadCount in quartz.properties (default is 10). But you cannot really do anything when the whole application/server/scheduler was down. The situation when Quartz was incapable of firing given trigger is called misfire. Do you know what Quartz is doing when it happens? Turns out there are various strategies (called misfire instructions) Quartz can take and also there are some defaults if you haven't thought about it. But in order to make your application robust and predictable (especially under heavy load or maintenance) you should really make sure your triggers and jobs are configured conciously. There are different configuration options (available misfire instructions) depending on the trigger chosen. Also Quartz behaves differently depending on trigger setup (so called smart policy). Although the misfire instructions are described in the documentation, I found it hard to understand what do they really mean. So I created this small summary article. Before I dive into the details, there is yet another configuration option that should be described. It is org.quartz.jobStore.misfireThreshold (in milliseconds), defaulting to 60000 (a minute). It defines how late the trigger should be to be considered misfired. With default setup if trigger was suppose to be fired 30 seconds ago, Quartz will happily just run it. Such delay is not considered misfiring. However if the trigger is discovered 61 seconds after the scheduled time - the special misfire handler thread takes care of it, obeying the misfire instruction. For test purposes we will set this parameter to 1000 (1 second) so that we can test misfiring quickly. Simple trigger without repeating In our first example we will see how misfiring is handled by simple triggers scheduled to run only once: val trigger = newTrigger(). startAt(DateUtils.addSeconds(new Date(), -10)). build() The same trigger but with explicitly set misfire instruction handler: val trigger = newTrigger(). startAt(DateUtils.addSeconds(new Date(), -10)). withSchedule( simpleSchedule(). withMisfireHandlingInstructionFireNow() //MISFIRE_INSTRUCTION_FIRE_NOW ). build() For the purpose of testing I am simply scheduling the trigger to run 10 seconds ago (so it is 10 seconds late by the time it is created!) In real world you would normally never schedule triggers like that. Instead imagine the trigger was set correctly but by the time it was scheduled the scheduler was down or didn't have any free worker threads. Nevertheless, how will Quartz handle this extraordinary situation? In the first code snippet above no misfire handling instruction is set (so called smart policy is used in that case). The second code snippet explicitly defines what kind of behaviour do we expect when misfiring occurs. See the table: Instruction Meaning smart policy - default See: withMisfireHandlingInstructionFireNow withMisfireHandlingInstructionFireNow MISFIRE_INSTRUCTION_FIRE_NOW The job is executed immediately after the scheduler discovers misfire situation. This is the smart policy. Example scenario: you have scheduled some system clean up at 2 AM. Unfortunately the application was down due to maintenance by that time and brought back on 3 AM. So the trigger misfired and the scheduler tries to save the situation by running it as soon as it can - at 3 AM. withMisfireHandlingInstructionIgnoreMisfires MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICY QTZ-283 See: withMisfireHandlingInstructionFireNow withMisfireHandlingInstructionNextWithExistingCount MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_EXISTING_COUNT See: withMisfireHandlingInstructionNextWithRemainingCount withMisfireHandlingInstructionNextWithRemainingCount MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_REMAINING_COUNT Does nothing, misfired execution is ignored and there is no next execution. Use this instruction when you want to completely discard the misfired execution. Example scenario: the trigger was suppose to start recording of a program in TV. There is no point of starting recording when the trigger misfired and is already 2 hours late. withMisfireHandlingInstructionNowWithExistingCount MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_EXISTING_REPEAT_COUNT See: withMisfireHandlingInstructionFireNow withMisfireHandlingInstructionNowWithRemainingCount MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_REMAINING_REPEAT_COUNT See: withMisfireHandlingInstructionFireNow Simple trigger repeating fixed number of times This scenario is much more complicated. Imagine we have scheduled some job to repeat fixed number of times: val trigger = newTrigger(). startAt(dateOf(9, 0, 0)). withSchedule( simpleSchedule(). withRepeatCount(7). withIntervalInHours(1). WithMisfireHandlingInstructionFireNow() //or other ). build() In this example the trigger is suppose to fire 8 times (first execution + 7 repetitions) every hour, beginning at 9 AM today (startAt(dateOf(9, 0, 0)). Thus the last execution should occur at 4 PM. However assume that due to some reason the scheduler was not capable of running jobs at 9 and 10 AM and it discovered that fact at 10:15 AM, i.e. 2 firings misfired. How will the scheduler behave in this situation? Instruction Meaning smart policy - default See: withMisfireHandlingInstructionNowWithExistingCount withMisfireHandlingInstructionFireNow MISFIRE_INSTRUCTION_FIRE_NOW See: withMisfireHandlingInstructionNowWithRemainingCount withMisfireHandlingInstructionIgnoreMisfires MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICYQTZ-283 Fires all triggers that were missed as soon as possible and then goes back to ordinary schedule. Example scenario: With this strategy in our example the scheduler will fire jobs scheduled at 9 and 10 AM immediately. Then it will wait to 11 AM and go back to ordinary schedule. Note: When handling misfires it is equally important to realize that the actual job execution time might be way after the scheduled time. This means you cannot simply rely on current system date, but you need to use JobExecutionContext .getScheduledFireTime(): def execute(context: JobExecutionContext) { val date = context.getScheduledFireTime //... } withMisfireHandlingInstructionNextWithExistingCount MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_EXISTING_COUNT The scheduler won't do anything immediately. Instead it will wait for next scheduled time and run all triggers with scheduled intervals. See also: withMisfireHandlingInstructionNextWithRemainingCount Example scenario: at 10:15 the scheduler discovers 2 misfired executions. It waits until next scheduled time (11 AM) and fires all 8 scheduled executions every hour, stopping at 6 PM (the trigger should have stopped at 4 PM). withMisfireHandlingInstructionNextWithRemainingCount MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_REMAINING_COUNT The scheduler discards misfired executions and waits for the next scheduled time. The total number of trigger executions will be less then configured. Example scenario: at 10:15 two misfired executions are discarded. The scheduler waits for next scheduled time (11 AM) and fires remaining triggers up to 4 PM. Effectively it behaves as if misfire never occurred. withMisfireHandlingInstructionNowWithExistingCount MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_EXISTING_REPEAT_COUNT First misfired trigger is executed immediately. Then the scheduler waits desired interval and executes all remaining triggers. Effectively the first fire time of the misfired trigger is moved to current time with no other changes. Example scenario: at 10:15 the scheduler runs the first misfired execution. Then it waits 1 hour and fires the second one at 11:15 AM. All 8 executions are performed, the last one at 5:15 PM withMisfireHandlingInstructionNowWithRemainingCount MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_REMAINING_REPEAT_COUNT First misfired execution runs immediately. Remaining misfired executions are discarded. Triggers that were not misfired are executed with desired interval. Example scenario: at 10:15 the scheduler runs the first misfired execution (from 9 AM). It discards remaining misfired executions (the one from 10 AM) and waits 1 hour to execute six more triggers: 11:15, 12:15, … 4:15 PM Simple trigger repeating infinitely In this scenario trigger repeats infinite number of times at a given interval: val trigger = newTrigger(). startAt(dateOf(9, 0, 0)). withSchedule( simpleSchedule(). withRepeatCount(SimpleTrigger.REPEAT_INDEFINITELY). withIntervalInHours(1). WithMisfireHandlingInstructionFireNow() //or other ). build() Once again trigger should fire on every hour, beginning at 9 AM today (startAt(dateOf(9, 0, 0)). However the scheduler was not capable of running jobs at 9 and 10 AM and it discovered that fact at 10:15 AM, i.e. 2 firings misfired. This is a more general situation compared to simple trigger running fixed number of times. Instruction Meaning smart policy - default See: withMisfireHandlingInstructionNextWithRemainingCount withMisfireHandlingInstructionFireNow MISFIRE_INSTRUCTION_FIRE_NOW See: withMisfireHandlingInstructionNowWithRemainingCount withMisfireHandlingInstructionIgnoreMisfires MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICYQTZ-283 The scheduler will immediately run all misfired triggers, then continue on schedule. Example scenario: the triggers scheduled at 9 and 10 AM are executed immediately. Future invocations (next scheduled at 11 AM) are executed according to the plan. withMisfireHandlingInstructionNextWithExistingCount MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_EXISTING_COUNT See: withMisfireHandlingInstructionNextWithRemainingCount withMisfireHandlingInstructionNextWithRemainingCount MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_REMAINING_COUNT Does nothing, misfired executions are discarded. Then the scheduler waits for next scheduled interval and goes back to schedule. Example scenario: Misfired execution at 9 and 10 AM are discarded. The first execution occurs at 11 AM. withMisfireHandlingInstructionNowWithExistingCount MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_EXISTING_REPEAT_COUNT See: withMisfireHandlingInstructionNowWithRemainingCount withMisfireHandlingInstructionNowWithRemainingCount MISFIRE_INSTRUCTION_RESCHEDULE_NOW_WITH_REMAINING_REPEAT_COUNT The first misfired execution is run immediately, remaining are discarded. Next execution happens after desired interval. Effectively the first execution time is moved to current time. Example scenario: the scheduler fires misfired trigger immediately at 10:15 AM. Then waits an hour and runs the second one at 11:15 AM and continues with 1 hour interval. CRON triggers CRON triggers are the most popular ones amongst Quartz users. However there are also two other available triggers: DailyTimeIntervalTrigger (e.g. fire every 25 minutes) and CalendarIntervalTrigger (e.g. fire every 5 months). They support triggering policies not possible in both CRON and simple triggers. However they understand the same misfire handling instructions as CRON trigger. val trigger = newTrigger(). withSchedule( cronSchedule("0 0 9-17 ? * MON-FRI"). withMisfireHandlingInstructionFireAndProceed() //or other ). build() In this example the trigger should fire every hour between 9 AM and 5 PM, from Monday to Friday. But once again first two invocations were missed (so the trigger misfired) and this situation was discovered at 10:15 AM. Note that available misfire instructions are different compared to simple triggers: Instruction Meaning smart policy - default See: withMisfireHandlingInstructionFireAndProceed withMisfireHandlingInstructionIgnoreMisfires MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICYQTZ-283 All misfired executions are immediately executed, then the trigger runs back on schedule. Example scenario: the executions scheduled at 9 and 10 AM are executed immediately. The next scheduled execution (at 11 AM) runs on time. withMisfireHandlingInstructionFireAndProceed MISFIRE_INSTRUCTION_FIRE_ONCE_NOW Immediately executes first misfired execution and discards other (i.e. all misfired executions are merged together). Then back to schedule. No matter how many trigger executions were missed, only single immediate execution is performed. Example scenario: the executions scheduled at 9 and 10 AM are merged and executed only once (in other words: the execution scheduled at 10 AM is discarded). The next scheduled execution (at 11 AM) runs on time. withMisfireHandlingInstructionDoNothing MISFIRE_INSTRUCTION_DO_NOTHING All misfired executions are discarded, the scheduler simply waits for next scheduled time. Example scenario: the executions scheduled at 9 and 10 AM are discarded, so basically nothing happens. The next scheduled execution (at 11 AM) runs on time. QTZ-283Note: QTZ-283: MISFIRE_INSTRUCTION_IGNORE_MISFIRE_POLICY not working with JDBCJobStore - apparently there is a bug when JDBCJobStore is used, keep an eye on that issue. As you can see various triggers behave differently based on the actual setup. Moreover, even though the so called smart policy is provided, often the decision is based on business requirements. Essentially there are three major strategies: ignore, run immediately and continue and discard and wait for next. They all have different use-cases: Use ignore policies when you want to make sure all scheduled executions were triggered, even if it means multiple misfired triggers will fire. Think about a job that generates report every hour based on orders placed during that last hour. If the server was down for 8 hours, you still want to have that reports generated, as soon as you can. In this case the ignore policies will simply run all triggers scheduled during that 8 hour as fast as scheduler can. They will be several hours late, but will eventually be executed. Use now* policies when there are jobs executing periodically and upon misfire situation they should run as soon as possible, but only once. Think of a job that cleans /tmp directory every minute. If the scheduler was busy for 20 minutes and finally can run this job, you don't want to run in 20 times! One is enough, but make sure it runs as fast it can. Then back to your normal one-minute intervals. Finally next* policies are good when you want to make sure your job runs at particular points in time. For example you need to fetch stock prices quarter past every hour. They change rapidly so if your job misfired and it is already 20 minutes past full hour, don't bother. You missed the correct time by 5 minutes and now you don't really care. It is better to have a gap rather than an inaccurate value. In this case Quartz will skip all misfired executions and simply wait for the next one.

i’ve done a few posts recently using d3.js and now i want to show you how to use two other great javascript libraries to visualize your graphs. we’ll start with sigma.js and soon i’ll do another post with three.js . we’re going to create our graph and group our nodes into five clusters. you’ll notice later on that we’re going to give our clustered nodes colors using rgb values so we’ll be able to see them move around until they find their right place in our layout. we’ll be using two sigma.js plugins, the gefx (graph exchange xml format) parser and the forceatlas2 layout. you can see what a gefx file looks like below. notice it comes from gephi which is an interactive visualization and exploration platform, which runs on all major operating systems, is open source, and is free. ... ... in order to build this file, we will need to get the nodes and edges from the graph and create an xml file. get '/graph.xml' do @nodes = nodes @edges = edges builder :graph end we’ll use cypher to get our nodes and edges: def nodes neo = neography::rest.new cypher_query = " start node = node:nodes_index(type='user')" cypher_query << " return id(node), node" neo.execute_query(cypher_query)["data"].collect{|n| {"id" => n[0]}.merge(n[1]["data"])} end we need the node and relationship ids, so notice i’m using the id() function in both cases. def edges neo = neography::rest.new cypher_query = " start source = node:nodes_index(type='user')" cypher_query << " match source -[rel]-> target" cypher_query << " return id(rel), id(source), id(target)" neo.execute_query(cypher_query)["data"].collect{|n| {"id" => n[0], "source" => n[1], "target" => n[2]} } end so far we have seen graphs represented as json, and we’ve built these manually. today we’ll take advantage of the builder ruby gem to build our graph in xml. xml.instruct! :xml xml.gexf 'xmlns' => "http://www.gephi.org/gexf", 'xmlns:viz' => "http://www.gephi.org/gexf/viz" do xml.graph 'defaultedgetype' => "directed", 'idtype' => "string", 'type' => "static" do xml.nodes :count => @nodes.size do @nodes.each do |n| xml.node :id => n["id"], :label => n["name"] do xml.tag!("viz:size", :value => n["size"]) xml.tag!("viz:color", :b => n["b"], :g => n["g"], :r => n["r"]) xml.tag!("viz:position", :x => n["x"], :y => n["y"]) end end end xml.edges :count => @edges.size do @edges.each do |e| xml.edge:id => e["id"], :source => e["source"], :target => e["target"] end end end end you can get the code on github as usual and see it running live on heroku. you will want to see it live on heroku so you can see the nodes in random positions and then move to form clusters. use your mouse wheel to zoom in, and click and drag to move around. credit goes out to alexis jacomy and mathieu jacomy . you’ve seen me create numerous random graphs, but for completeness here is the code for this graph. notice how i create 5 clusters and for each node i assign half its relationships to other nodes in their cluster and half to random nodes? this is so the forceatlas2 layout plugin clusters our nodes neatly. def create_graph neo = neography::rest.new graph_exists = neo.get_node_properties(1) return if graph_exists && graph_exists['name'] names = 500.times.collect{|x| generate_text} clusters = 5.times.collect{|x| {:r => rand(256), :g => rand(256), :b => rand(256)} } commands = [] names.each_index do |n| cluster = clusters[n % clusters.size] commands << [:create_node, {:name => names[n], :size => 5.0 + rand(20.0), :r => cluster[:r], :g => cluster[:g], :b => cluster[:b], :x => rand(600) - 300, :y => rand(150) - 150 }] end names.each_index do |from| commands << [:add_node_to_index, "nodes_index", "type", "user", "{#{from}"] connected = [] # create clustered relationships members = 20.times.collect{|x| x * 10 + (from % clusters.size)} members.delete(from) rels = 3 rels.times do |x| to = members[x] connected << to commands << [:create_relationship, "follows", "{#{from}", "{#{to}"] unless to == from end # create random relationships rels = 3 rels.times do |x| to = rand(names.size) commands << [:create_relationship, "follows", "{#{from}", "{#{to}"] unless (to == from) || connected.include?(to) end end batch_result = neo.batch *commands end

Although briefly described in the official documentation, I believe Quartz plugins aren't known enough, looking at how useful they are. Essentially plugins in Quartz are convenient classes wrapping registration of underlying listeners. You are free to write your own plugins but we will focus on existing ones shipped with Quartz. LoggingTriggerHistoryPlugin First some background. Two main abstractions in Quartz are jobs and triggers. Job is a piece of code that we would like to schedule. Trigger instructs the scheduler when this code should run. CRON (e.g. run every Friday between 9 AM and 5 PM until November) and simple (run 100 times every 2 hours) triggers are most commonly used. You associate any number of triggers to a single job. Believe it or not, Quartz by default provides no logging or monitoring whatsoever of executed jobs and triggers. There is an API, but no built-in logging is implemented. It won't show you that it now executes this particular job due to this trigger firing. So the first thing you should do is adding the following lines to your quartz.properties: org.quartz.plugin.triggerHistory.class=org.quartz.plugins.history.LoggingTriggerHistoryPlugin org.quartz.plugin.triggerHistory.triggerFiredMessage=Trigger [{1}.{0}] fired job [{6}.{5}] scheduled at: {2, date, dd-MM-yyyy HH:mm:ss.SSS}, next scheduled at: {3, date, dd-MM-yyyy HH:mm:ss.SSS} org.quartz.plugin.triggerHistory.triggerCompleteMessage=Trigger [{1}.{0}] completed firing job [{6}.{5}] with resulting trigger instruction code: {9}. Next scheduled at: {3, date, dd-MM-yyyy HH:mm:ss.SSS} org.quartz.plugin.triggerHistory.triggerMisfiredMessage=Trigger [{1}.{0}] misfired job [{6}.{5}]. Should have fired at: {3, date, dd-MM-yyyy HH:mm:ss.SSS} The first line (and the only required) loads the plugin class LoggingTriggerHistoryPlugin. The remaining lines are configuring the plugin, customizing the logging messages. I found the built-in defaults not very well thought, e.g. they display current time which is already part of the logging framework message. You are free to construct any logging message, see the API for details. Adding these extra few lines makes debugging and monitoring much easier: LoggingTriggerHistoryPlugin | Trigger [Demo.Every-few-seconds] fired job [Demo.Print-message] scheduled at: 04-04-2012 23:23:47.036, next scheduled at: 04-04-2012 23:23:51.036 //...job output LoggingTriggerHistoryPlugin | Trigger [Demo.Every-few-seconds] completed firing job [Demo.Print-message] with resulting trigger instruction code: DO NOTHING. Next scheduled at: 04-04-2012 23:23:51.036 You see now why naming your triggers (Demo.Every-few-seconds) and jobs (Demo.Print-message) is so important. LoggingJobHistoryPlugin There is another handy plugin related to logging: org.quartz.plugin.jobHistory.class=org.quartz.plugins.history.LoggingJobHistoryPlugin org.quartz.plugin.jobHistory.jobToBeFiredMessage=Job [{1}.{0}] to be fired by trigger [{4}.{3}], re-fire: {7} org.quartz.plugin.jobHistory.jobSuccessMessage=Job [{1}.{0}] execution complete and reports: {8} org.quartz.plugin.jobHistory.jobFailedMessage=Job [{1}.{0}] execution failed with exception: {8} org.quartz.plugin.jobHistory.jobWasVetoedMessage=Job [{1}.{0}] was vetoed. It was to be fired by trigger [{4}.{3}] at: {2, date, dd-MM-yyyy HH:mm:ss.SSS} The rule is the same - plugin + extra configuration. See JavaDoc of LoggingJobHistoryPlugin for details and possible placeholders. Quick look at logs reveals very descriptive output: Trigger [Demo.Every-few-seconds] fired job [Demo.Print-message] scheduled at: 04-04-2012 23:34:53.739, next scheduled at: 04-04-2012 23:34:57.739 Job [Demo.Print-message] to be fired by trigger [Demo.Every-few-seconds], re-fire: 0 //...job output Job [Demo.Print-message] execution complete and reports: null Trigger [Demo.Every-few-seconds] completed firing job [Demo.Print-message] with resulting trigger instruction code: DO NOTHING. Next scheduled at: 04-04-2012 23:34:57.739 I have no idea why these plugins aren't enabled by default. After all, if you don't want such a verbose output, you can turn it off in your logging framework. Never mind, I think it is a good idea to have them in place when troubleshooting Quartz execution. XMLSchedulingDataProcessorPlugin This is a pretty comprehensive plugin. It reads XML file (by default named quartz_data.xml) containing jobs and triggers definitions and adds them to the scheduler. This is especially useful when you have a global job that you need to add once. Plugin can either update the existing jobs/triggers or ignore the XML file if they already exist - very useful when JDBCJobStore is used. org.quartz.plugin.xmlScheduling.class=org.quartz.plugins.xml.XMLSchedulingDataProcessorPlugin In the aforementioned article we have been manually adding job to the scheduler: val trigger = newTrigger(). withIdentity("Every-few-seconds", "Demo"). withSchedule( simpleSchedule(). withIntervalInSeconds(4). repeatForever() ). build() val job = newJob(classOf[PrintMessageJob]). withIdentity("Print-message", "Demo"). usingJobData("msg", "Hello, world!"). build() scheduler.scheduleJob(job, trigger) The same can be achieved with XML configuration, just place the following quartz_data.xml in your CLASSPATH: false true Every-few-seconds Demo Print-message Demo -1 4000 Print-message Demo com.blogspot.nurkiewicz.quartz.demo.PrintMessageJob msg Hello, World! The file supports both simple and CRON triggers and is well described using XML Schema. It is even possible to point out to an XML files somewhere in the file system and periodically scan them for changes (!) (see: XMLSchedulingDataProcessorPlugin.setScanInterval(). Guess what is Quartz using to schedule periodic scanning? org.quartz.plugin.xmlScheduling.fileNames=/etc/quartz/system-jobs.xml,/home/johnny/my-jobs.xml org.quartz.plugin.xmlScheduling.scanInterval=60 ShutdownHookPlugin Last but not least, ShutdownHookPlugin. Small but probably useful plugin that register shutdown hook in the JVM in order to gently stop the scheduler. However I recommend turning cleanShutdown off - if the system already tries to abruptly stop the application (typically scheduler shutdown is called by Spring via SchedulerFactoryBean) or the user hit Ctrl+C - waiting for currently running jobs seems like a bad idea. After all, maybe we are killing the application because some jobs are running for too long/hanging? org.quartz.plugin.shutdownHook.class=org.quartz.plugins.management.ShutdownHookPlugin org.quartz.plugin.shutdownHook.cleanShutdown=false As you can see Qurtz ships with few quite interesting plugins. For some reason they aren't described in detail in the official documentation, but they work pretty well and are a valuable addition to scheduler. The source code with applied plugins is available on GitHub.

I am starting a little series about Quartz scheduler internals, tips and tricks, this is chapter 0 - how to configure persistent job store.

Since the very early days of Java (and C-like languages overall), the canonical way to start your program has been something like this: public class A { public static void main(String[] args) { new A().run(args); } public void run(String[] args) { // Your application starts here } } If you are still doing this, I’m here to tell you it’s time to stop. Letting go of ‘new’ First, install Guice in your project: com.google.inject guice 3.0 and then, modify your main method as follows: public class A { public static void main(String[] args) { Injector.getInstance(A.class).run(args); } } So, what does this buy you exactly? You will find a lot of articles explaining the various benefits of Guice, such as being able to substitute different environments on the fly, but I’m going to use a different angle in this article. Let’s start by assuming the existence of a Config class that contains various configuration parameters. I’ll just hardcode them for now and use fields to make the class smaller: public class Config { String host = "com.example.com"; int port = 1234; } This class is a singleton, it is instantiated somewhere in your main class and not used anywhere else at the moment. One day, you realize you need this instance in another class which happens to be deep in your runtime hierarchy, which we will call Deep. For example, if you put a break point in the method where you need this config object, your debugger would show you stack frames similar to this: com.example.A.main() com.example.B.f(int, String) com.example.C.g(String) com.example.Deep.h(Foo, int) The easy and wrong way to solve this problem is to make the Config instance static on some class (probably A) and access it directly from Deep. I’m hoping I don’t need to explain why this is a bad idea: not only do you want to avoid using statics, but you also want to make sure that each object is exposed only to objects that need them, and making the Config object static would make your instance visible to your entire code base. Not a good thing. The second thought is to pass the object down the stack, so you modify all the signatures as follows: com.example.A.main() com.example.B.f(int, String, Config) com.example.C.g(String, Config) com.example.Deep.h(Foo, int, Config) This is a bit better since you have severely restricted the exposure of the Config object, but note that you are still making it available to more methods than really need to: B#f and C#g have really nothing to do with this object, and a little sting of discomfort hits you when you start writing the Javadoc: public class C { ... /** * @param config This method doesn't really use this parameter, * it just passes it down so Deep#h can use it. */ public void g(String s, Config config) { Unnecessary exposure is actually not the worst part of this approach, the problem is that it changes all these signatures along the way, which is certainly undesirable in a private API and absolutely devastating in a public API. And of course, it’s absolutely not scalable: if you keep adding a parameter to your method whenever you need access to a certain object, you will soon be dealing with methods that take ten parameters, most of which they just pass down the chain. Here is how we solve this problem with dependency injection (performed by Guice in this example, but this is applicable to any library that implements JSR 330, obviously): public class Deep { @Inject private Config config; and we’re done. That’s it. You don’t need to modify the Config class in any way, nor do you need to make any change in any of the classes that separate Deep from your main class. With this, you have also minimized the exposure of the Config object to just the class that needs it. Injecting right There are various ways you can inject object into your class but I’ll just mention the two that, I think, are the most important. I just showed “field injection” in the previous paragraph, but be aware that you can also prefer to use “constructor injection”: public class Deep { private final Config config; @Inject public Deep(Config config) { this.config = config; } This time, you are adding a parameter to the constructor of your Deep class (which shouldn’t worry you too much since you will never invoke it directly, Guice will) and you assign the parameter to the field in the constructor. The benefit is that you can declare your field final. The downside, obviously, is that this approach is much more verbose. Personally, I see little point in final fields since I have hardly ever encountered a bug that was due to accidentally reassigning a field, so I tend to use field injection whenever I can. Taking it to the next level Obviously, the kind of configuration object I used as an example if not very realistic. Typically, a configuration will not hardcode values like I did and will, instead, read them from some external source. Similarly, you will want to inject objects that can’t necessarily be instantiated so early in the lifecycle of your application, such as servlet contexts, database connections, or implementations of your own interfaces. This topic itself would probably cover several chapters of a book dedicated to dependency injection, so I’ll just summarize it: not all objects can be injected this way, and one benefit of using a dependency injection framework in your code is that it will force you to think about what life cycle category your objects belong to. Having said that, if you want to find out how Guice can inject objects that get created at a later time in your application life cycle, look up the Javadoc for the Provider class. Wrapping up I hope this quick introduction to dependency injection piqued your interest and that you will consider using it in your project since it has so much more to offer than what I described in this post. If you want to learn more, I suggest starting with the excellent Guice documentation.

JMS providers like ActiveMQ are based on the concept of passing one-directional messages between nodes and brokers asynchronously. A thorough knowledge of the type of messages that can be sent through a JMS middleware can simplify a lot your work in mapping the communication patterns to real code. The basic Message interface Some object members are shared by all messages: header fields, used to identify univocally a message and to route it to the right brokers and consumers. A dynamic map of properties which can be read programmatically by JMS brokers in order to filter or to route messages. A body, which is differentiated in the various implementations we'll see. Header fields The set of getJMS*() methods on the Message interface defines the available headers. Two of them are oriented to message identification: getJMSMessageID() contains a generated ID for identifying a message, unique at least for the current broker. All generated IDs start with the prefix 'ID:', but you can override it with the corresponding setter. getJMSCorrelationID() (and getJMSCorrelationID() as bytes) can link a message with another, usually one that has been sent previously. For example, a reply can carry the ID of the original message when put in another queue. Two to sender and recipient identification: getJMSDestination() returns a Destination object (a Topic or a Queue, or their temporary version) describing where the message was directed. getJMSReplyTo() is a Destination object where replies should be sent; it can be null of course. Three tune the delivery mechanism: getJMSDeliveryMode() can be DeliveryMode.NON_PERSISTENT or DeliveryMode.PERSISTENT; only persistent messages guarantee delivery in case of a crash of the brokers that transport it. getJMSExpiration() returns a timestamp indicating the expiration time of the message; it can be 0 on a message without a defined expiration. getJMSPriority() returns a 0-9 integer value (higher is better) defining the priority for delivery. It is only a best-effort value. While the remaining ones contain metadata: getJMSRedelivered() returns a boolean indicating if the message is being delivered again after a delivery which was not acknowledge. getJMSTimestamp() returns a long indicating the time of sending. getJMSType() defines a field for provider-specific or application-specific message types. Of these headers, only JMSCorrelationID, JMSReplyTo and JMSType have to be set when needed. The others are generated or managed by the send() and publish() methods if not specified (there are setters available for each of these headers.) Properties Generic properties with a String name can be added to messages and read with getBooleanProperty(), getStringProperty() and similar methods. The corresponding setters setBooleanProperty(), setStringProperty(), ... can be used for their addition. The reason for keeping some properties out of the content of the message (which a MapMessage can contain) is so they could be read before reaching the destination, for example in a JMS broker. The use case for this access to message properties is routing and filtering: downstream brokers and consumers may define a filter such as "I am interested only on messages on this Topic that have property X = 'value' and Y = 'value2'". Bodies All the subinterfaces of javax.jms.Message defined by the API provide different types of message bodies (while actual classes are defined by the providers and are not part of the API). Actual instantiation is then handled by the Session, which implements an Abstract Factory pattern. On the receival side, a cast is necessary for any message type (at least in the Java JMS Api), since only a generic Message is read. BytesMessage is the most basic type: it contains a sequence of uninterpreted bytes. Hence, it can in theory contain anything, but the generation and interpretation of the content is the client's job. BytesMessage m = session.createBytesMessage(); m.writeByte(65); m.writeBytes(new byte[] { 66, 68, 70 }); // on receival (cast shown only here) BytesMessage m = (BytesMessage) genericMessage; byte[] content = new byte[4]; m.readBytes(content); MapMessage defines a message containing an (unordered) set of key/value pairs, also called a map or dictionary or hash. However, the keys are String objects, while the values are primitives or Strings; since they are primitives, they shouldn't be null. MapMessage = session.createMapMessage(); m.setString('key', 'value'); // or m.setObject('key', 'value') to avoid specifying a type // on receival m.getString('key'); // or m.getObject('key') ObjectMessage wraps a generic Object for transmission. The Object should be Serializable. ObjectMessage m = session.createObjectMessage(); m.setObject(new ValueObject('field1', 42)); // on receival ValueObject vo = (ValueObject) m.getObject(); StreamMessage wraps a stream of primitive values of indefinite length. StreamMessage m = session.createStreamMessage(); m.writeBoolean(true); m.writeBoolean(false); m.writeBoolean(true); // receival System.out.println(m.readBoolean()); System.out.println(m.readBoolean()); System.out.println(m.readBoolean()); // prints true, false, true TextMessage wraps a String of any length. TextMessage m = session.createTextMessage("Contents"); // or use m.setText() afterwards // receival String text = m.getText(); Usually all messages are in a read-only phase after receival, so only getters can be called on them.

My prefered solution was to use the Maven ‘update snapshots’ command line argument.

We’ve been trying to grow our team for a few months now and the title we’re hiring for is Devops Engineer. One of the candidates our recruiters reached out to, let’s call him John, came back to us with a bunch of questions including: How do you feel about hiring someone with a devops title? It’s a very legittimate question, Devops is a cultural and professional movement, so how could it be a job title? What I argued in my reply to this fella is that Devops isn’t the job title, Devops Engineer is, and in this sense Devops is just a qualifier and I strongly believe a very useful one. I really sympathise with those that are fighting hard to keep Devops real and avoid the same faith that some refer to as the sad commercialisation of Agile. My campaign to make of devops a job title isn’t a campaign to come up with a set of bullet points that define Devops as a job so that I can put it on a resume or build it into a product. My argument here is that the guy I’m trying to hire, John, I want him to be a certain kind of guy and the best way I have to describe what I want is Devops Engineer. I’m looking for an operations guy , but I want him to be open to developers, consider engineering and the company as a whole, be focused on delivering value and not rathole into fights about technology or claim root access only on principle. I want that guy to have great communication skills and the interest to explore what’s besides his infrastructure, to be wanting to borrow as much good he can find in other disciplines across the organisation. And then of course there is the practical part, the desire to automate and escape a boring manual routine, the familiarity with cloud that willing or not has powered the movement, and even more specific things like configuration management. You may argue that this is just a good engineer or what systems engineers are becoming, in other words nothing new under the sun. And you may be right, but job titles are in many ways just another way to communicate, to broadcast an intent and a need. So you know what I told John about hiring Devops Engineers? That I felt pretty damn proud about it. The true ones, not the ones slapping it on their CV to get a job, are fantastic engineers and I can’t but encourage them to start to respond to that qualifier. Likewise the companies and individuals seeking them out are likely the ones building great groups those people will want to be members of. Yes, the moment it becomes a keyword recruiters start to match against we’re likely to see a spur of fakes trying to land a job, but that’s nothing new under the sun. Signed, a Devops manager Source: http://www.spikelab.org/devops-job-title/

a pre-commit hook is a piece of code that runs before every commit and determines whether or not the commit should be accepted. think of it as the gatekeeper to your codebase. want to ensure you didn’t accidentally leave any pdb s in your code? pre-commit hook. want to make sure your javascript is jshint approved? pre-commit hook. want to guarantee clean, readable pep8 -compliant code? pre-commit hook. want to pipe all of the comments in your codebase through strunk & white ? please don’t. the pre-commit hook is just an executable file that runs before every commit. if it exits with zero status, the commit is accepted. if it exits with a non-zero status, the commit is rejected. (note: a pre-commit hook can be bypassed by passing the --no-verify argument.) along with the pre-commit hook there are numerous other git hooks that are available: post-commit, post-merge, pre-receive, and others that can be found here . why most pre-commit hooks are wrong be wary of the above’s example as the majority of pre-commit hooks you’ll see on the web are wrong. most test against whatever files are currently on disk, not what is in the staging area (the files actually being committed). we avoid this in our hook by stashing all changes that are not part of the staging area before running our checks and then popping the changes afterwards. this is very important because a file could be fine on disk while the changes that are being committed are wrong. the code below is the pre-commit hook we use at yipit. our hook is simply a set of checks to be run against any files that have been modified in this commit. each check can be configured to include/exclude particular types of files. it is designed for a django environment, but should be adaptable to other environments with minor changes. note that you need git 1.7.7+ #!/usr/bin/env python import os import re import subprocess import sys modified = re.compile('^(?:m|a)(\s+)(?p.*)') checks = [ { 'output': 'checking for pdbs...', 'command': 'grep -n "import pdb" %s', 'ignore_files': ['.*pre-commit'], 'print_filename': true, }, { 'output': 'checking for ipdbs...', 'command': 'grep -n "import ipdb" %s', 'ignore_files': ['.*pre-commit'], 'print_filename': true, }, { 'output': 'checking for print statements...', 'command': 'grep -n print %s', 'match_files': ['.*\.py$'], 'ignore_files': ['.*migrations.*', '.*management/commands.*', '.*manage.py', '.*/scripts/.*'], 'print_filename': true, }, { 'output': 'checking for console.log()...', 'command': 'grep -n console.log %s', 'match_files': ['.*yipit/.*\.js$'], 'print_filename': true, }, { 'output': 'checking for debugger...', 'command': 'grep -n debugger %s', 'match_files': ['.*\.js$'], 'print_filename': true, }, { 'output': 'running jshint...', # by default, jshint prints 'lint free!' upon success. we want to filter this out. 'command': 'jshint %s | grep -v "lint free!"', 'match_files': ['.*yipit/.*\.js$'], 'print_filename': false, }, { 'output': 'running pyflakes...', 'command': 'pyflakes %s', 'match_files': ['.*\.py$'], 'ignore_files': ['.*settings/.*', '.*manage.py', '.*migrations.*', '.*/terrain/.*'], 'print_filename': false, }, { 'output': 'running pep8...', 'command': 'pep8 -r --ignore=e501,w293 %s', 'match_files': ['.*\.py$'], 'ignore_files': ['.*migrations.*'], 'print_filename': false, }, { 'output': 'checking for sass changes...', 'command': 'sass --quiet --update %s', 'match_files': ['.*\.scss$'], 'print_filename': true, }, ] def matches_file(file_name, match_files): return any(re.compile(match_file).match(file_name) for match_file in match_files) def check_files(files, check): result = 0 print check['output'] for file_name in files: if not 'match_files' in check or matches_file(file_name, check['match_files']): if not 'ignore_files' in check or not matches_file(file_name, check['ignore_files']): process = subprocess.popen(check['command'] % file_name, stdout=subprocess.pipe, stderr=subprocess.pipe, shell=true) out, err = process.communicate() if out or err: if check['print_filename']: prefix = '\t%s:' % file_name else: prefix = '\t' output_lines = ['%s%s' % (prefix, line) for line in out.splitlines()] print '\n'.join(output_lines) if err: print err result = 1 return result def main(all_files): # stash any changes to the working tree that are not going to be committed subprocess.call(['git', 'stash', '-u', '--keep-index'], stdout=subprocess.pipe) files = [] if all_files: for root, dirs, file_names in os.walk('.'): for file_name in file_names: files.append(os.path.join(root, file_name)) else: p = subprocess.popen(['git', 'status', '--porcelain'], stdout=subprocess.pipe) out, err = p.communicate() for line in out.splitlines(): match = modified.match(line) if match: files.append(match.group('name')) result = 0 print 'running django code validator...' return_code = subprocess.call('$virtual_env/bin/python manage.py validate', shell=true) result = return_code or result for check in checks: result = check_files(files, check) or result # unstash changes to the working tree that we had stashed subprocess.call(['git', 'reset', '--hard'], stdout=subprocess.pipe, stderr=subprocess.pipe) subprocess.call(['git', 'stash', 'pop', '-q'], stdout=subprocess.pipe, stderr=subprocess.pipe) sys.exit(result) if __name__ == '__main__': all_files = false if len(sys.argv) > 1 and sys.argv[1] == '--all-files': all_files = true main(all_files) to use this hook or a hook that you create yourself, simply copy the file to .git/hooks/pre-commit inside of your project and make sure that it is executable or add in to your git repo and setup a symlink.

today i have prepared another really great tutorial for you. recently i started development of my own radio software (as a module for dolphin cms) and got some interesting results i would like to share with you. it will be nice looking (css3) radio script that consists of three main elements: the header (with animated search bar and integrated radio player), the left side (with a list of categories and subcategories) and the right side (which will contain a list of recent or filtered stations). here's the final result of our player: here's our live demo and downloadable package: ok, download our source files and lets start coding! step 1. html markup this is the markup of one of the template files. this is a template of our main (index) page: templates/main_page.html stream radio script back to original tutorial on script tutorials alternative classic alternativeindustrialnew wavepunk classical modernoperapianoromanticsymphony electronic breakbeatdanceelectrohousetechnotrance metal classic metalheavy metalmetalcorepower metal pop dance popoldiestop 40world pop __stations__ powered by script tutorials first, pay attention to how the script loads the jquery library from google. this can be pretty useful if you don’t like to keep this file directly on your host. our header element contains a nice search bar with an embedded jasl player ( i used a great ffmp3 live stream player ), which allows us to play audio streams without any problems. next, on the left-hand side (beneath the header) we have a ul-li based list of categories and subcategories. the right-hand side will contain a list of the most recent stations and, when we search or select a category, the right-hand side will be filtered by ajaxy. for now – it contains __stations__ key (template key) and we will replace the actual value with php. on to our next template file, the radio player: templates/radio.html of course, it contains its own template keys (__title__ and __stream__) which we will use after. step 2. css here are our stylesheets files: css/main.css the first one contains the styles of our test page (this file is always available in our package) css/radio.css /* header area */ .header { height:62px; } .header input { background:#aaa url(../images/search.png) no-repeat 5px center; border:1px solid #888; border-radius:10px; float:right; margin:14px 10px 0 0; outline:none; padding-left:20px; width:200px; -webkit-transition: 0.5s; -moz-transition: 0.5s; -o-transition: 0.5s; transition: 0.5s; } .header input:focus { background-color:#eee; width:300px; } .header > span { display:block; float:left; line-height:40px; padding:7px; -webkit-transition: 0.5s; -moz-transition: 0.5s; -o-transition: 0.5s; transition: 0.5s; } /* stations list */ .stlist { float:right; margin-right:1%; width:71%; } .stlist ul { list-style:none outside none; margin:0; padding:0; } .stlist ul li { border-bottom:1px dotted #444; overflow:hidden; padding:10px; } .stlist ul li > a > img { border:1px solid #ccc; float:left; height:85px; margin-right:15px; padding:1px; width:85px; } .stlist ul li > div { float:right; margin-left:15px; margin-top:-5px; } .stlist ul li > p.label,.stlist ul li > p.track { font-size:11px; font-weight:700; } .stlist ul li > p.label { color:#888; } .stlist ul li > p.channel { font-size:14px; font-weight:700; margin-bottom:17px; } /* genres list */ .genres_par { border-right:1px solid #ccc; float:left; width:26%; } ul.genres,ul.genres ul { list-style-type:none; margin:0; padding:0; } ul.genres ul { display:none; overflow:hidden; padding:0 15px; } ul.genres ul li { margin:3px; } ul.genres a { color:#333; display:block; font-size:18px; padding:4px 0; text-align:center; text-decoration:none; } ul.genres ul a { font-size:12px; text-align:left; } ul.genres li { border-bottom:1px solid #ccc; margin:0; } ul.genres li ul li a { background:none repeat scroll 0 0 #5bb951; border-radius:2px; color:#fff; font-size:12px; padding:6px; } ul.genres li ul li a:hover { background-color:#53854e; } step 3. js js/script.js $(document).ready(function(){ $('#search').blur(function() { if ('' == $('#search').val()) $('#search').val('search'); }); $('#search').focus(function() { if ('search' == $('#search').val()) $('#search').val(''); }); $('ul.genres li a').click( // category slider function() { var checkelement = $(this).next(); if((checkelement.is('ul')) && (!checkelement.is(':visible'))) { $('.genres li ul').slideup(150); $(this).next().slidetoggle(150); } } ); $('ul.genres ul li a').click( // get stations by category function() { $.ajax({ type: 'get', url: 'index.php', data: 'action=get_genre_stations&id=' + $(this).parent().attr('id') + '&name=' + $(this).parent().attr('val'), success: function(data){ $('.stlist').fadeout(400, function () { $('.stlist').html(data); $('.stlist').fadein(400); }); } }); } ); }); function play(id) { // play function $('#rplayer').load('index.php?action=play&id=' + id, function() {}); return false; } function get_stations_by_keyword() { // get stations by keyword var keyword = $('#search').val().replace(/ /g,"+"); $.ajax({ type: 'get', url: 'index.php', data: 'action=get_keyword_stations&key=' + keyword, success: function(data){ $('.stlist').fadeout(400, function () { $('.stlist').html(data); $('.stlist').fadein(400); }); } }); } as you see – there's nothing difficult there. just several event handlers, and two new functions (to play radio station and to search for stations by keyword). step 4. php index.php =') == 1) error_reporting(e_all & ~e_notice & ~e_deprecated); else error_reporting(e_all & ~e_notice); $astations = array( 0 => array( 'category' => 31, 'name' => 'eurodance', 'desc' => 'the newest and best of eurodance hits', 'url' => 'http://www.di.fm/eurodance', 'br' => 96, 'stream' => 'http://scfire-mtc-aa06.stream.aol.com:80/stream/1024' ), 1 => array ( 'category' => 34, 'name' => 'house', 'desc' => 'silky sexy deep house music direct from new york city!', 'url' => 'http://www.di.fm/house', 'br' => 96, 'stream' => 'http://scfire-ntc-aa04.stream.aol.com:80/stream/1007' ), 2 => array ( 'category' => 13, 'name' => 'trance', 'desc' => 'the hottest, freshest trance music from around the globe!', 'url' => 'http://www.di.fm/trance', 'br' => 96, 'stream' => 'http://scfire-ntc-aa04.stream.aol.com:80/stream/1003' ), 3 => array ( 'category' => 51, 'name' => 'electro house', 'desc' => 'an eclectic mix of electro and dirty house', 'url' => 'http://www.di.fm/electro', 'br' => 96, 'stream' => 'http://scfire-ntc-aa04.stream.aol.com:80/stream/1025' ) ); function searchbycat($icat, $astations) { $ares = array(); foreach ($astations as $i => $ainfo) { if ($ainfo['category'] == $icat) { $ares[$i] = $ainfo; } } return $ares; } function searchbykeyword($skey, $astations) { $ares = array(); foreach ($astations as $i => $ainfo) { if (false !== strpos($ainfo['name'], $skey) || false !== strpos($ainfo['desc'], $skey)) { $ares[$i] = $ainfo; } } return $ares; } function parsestationlist($adata) { $sstations = ''; if (is_array($adata) && count($adata) > 0) { foreach ($adata as $i => $a) { $sstationid = $i; $sstationbr = (int)$a['br']; $sstationname = $a['name']; $sstationdesc = $a['desc']; $sstationurl = $a['url']; $sthumb = 'media/'.($sstationid+1).'.png'; $sstations .= << bitrate: {$sstationbr} {$sstationname} {$sstationdesc} {$sstationurl} eof; } } $sstations = ($sstations == '') ? 'nothing found' : $sstations; return '' . $sstations . ''; } switch ($_get['action']) { case 'play': $i = (int)$_get['id']; $ainfo = $astations[$i]; $avars = array ( '__stream__' => $ainfo['stream'], '__title__' => $ainfo['name'] ); echo strtr(file_get_contents('templates/radio.html'), $avars); exit; break; case 'get_genre_stations': $i = (int)$_get['id']; $asearch = searchbycat($i, $astations); $sstations = parsestationlist($asearch); header('content-type: text/html; charset=utf-8'); echo $sstations; exit; break; case 'get_keyword_stations': $skey = $_get['key']; $asearch = searchbykeyword($skey, $astations); $sstations = parsestationlist($asearch); header('content-type: text/html; charset=utf-8'); echo $sstations; exit; break; } $slaststations = parsestationlist($astations); echo strtr(file_get_contents('templates/main_page.html'), array('__stations__' => $slaststations)); at the beginning, i have prepared a list of our radio stations (4 stations total). then, two search functions: ‘searchbycat’ and ‘searchbykeyword’. next, the special function ‘parsestationlist’ which will transform array with filtered stations into its html representation. finally, a little switch case to manage with our inner ajax commands. conclusion you are always welcome to enhance our script and share your ideas. i will be glad to see your thanks and comments. good luck!