An increasing number of systems are being built to handle the Volume, Velocity and Variety of Big Data, and hopefully help gain new insights and make better business decisions. Here, we will look at ways to deal with Big Data’s Volume and Velocity simultaneously, within a single architecture solution. Volume + Velocity Apache Hadoop provides both reliable storage (HDFS) and a processing system (MapReduce) for large data sets across clusters of computers. MapReduce is a batch query processor that is targeted at long-running background processes. Hadoop can handle Volume. But to handle Velocity, we need real-time processing tools that can compensate for the high-latency of batch systems, and serve the most recent data continuously, as new data arrives and older data is progressively integrated into the batch framework. Therefore we need both batch and real-time to run in parallel, and add a real-time computational system (e.g. Apache Storm) to our batch framework. This architectural combination of batch and real-time computation is referred to as a Lambda Architecture (λ). Generic Lambda λ has three layers: The Batch Layer manages the master data and precomputes the batch views The Speed Layer serves recent data only and increments the real-time views The Serving Layer is responsible for indexing and exposing the views so that they can be queried. The three layers are outlined in the below diagram along with a sample choice of technology stacks: Incoming data is dispatched to both Batch and Speed layers for processing. At the other end, queries are answered by merging both batch and real-time views. Note that real-time views are transient by nature and their data is discarded (making room for newer data) once propagated through the Batch and Serving layers. Most of the complexity is pushed onto the much smaller Speed layer where the results are only temporary, a process known as “complexity isolation“. We are indeed isolating the complexity of concurrent data updates in a layer that is regularly purged and kept small in size. λ is technology agnostic. The data pipeline is broken down into layers with clear demarcation of responsibilities, and at each layer, we can choose from a number of technologies. The Speed layer for instance could use either Apache Storm, or Apache Spark Streaming, or Spring “XD” ( eXtreme Data) etc. How do we recover from mistakes in λ ? Basically, we recompute the views. If that takes too long, we just revert to the previous, non-corrupted versions of our data. We can do that because of data immutability in the master dataset: data is never updated, only appended to (time-based ordering). The system is therefore Human Fault-Tolerant: if we write bad data, we can just remove that data altogether and recompute. Unified Lambda The downside of λ is its inherent complexity. Keeping in sync two already complex distributed systems is quite an implementation and maintenance challenge. People have started to look for simpler alternatives that would bring just about the same benefits and handle the full problem set. There are basically three approaches: 1) Adopt a pure streaming approach, and use a flexible framework such as Apache Samza to provide some type of batch processing. Although its distributed streaming layer is pluggable, Samza typically relies on Apache Kafka. Samza’s streams are replayable, ordered partitions. Samza can be configured for batching, i.e. consume several messages from the same stream partition in sequence. 2) Take the opposite approach, and choose a flexible Batch framework that would also allow micro-batches, small enough to be close to real-time, with Apache Spark/Spark Streaming or Storm’s Trident. Spark streaming is essentially a sequence of small batch processes that can reach latency as low as one second.Trident is a high-level abstraction on top of Storm that can process streams as small batches as well as do batch aggregation. 3) Use a technology stack already combining batch and real-time, such as Spring “XD”, Summingbird or Lambdoop. Summingbird (“Streaming MapReduce”) is a hybrid system where both batch/real-time workflows can be run at the same time and the results merged automatically.The Speed layer runs on Storm and the Batch layer on Hadoop, Lambdoop (Lambda-Hadoop, with HBase, Storm and Redis) also combines batch/real-time by offering a single API for both processing paradigms: The integrated approach (unified λ) seeks to handle Big Data’s Volume and Velocity by featuring a hybrid computation model, where both batch and real-time data processing are combined transparently. And with a unified framework, there would be only one system to learn, and one system to maintain.

In this tip you can see how to use the Angular Websocket module for connecting client applications to servers.

Want to add a feature or automate something in your NetBeans IDE? Follow along as we write your first plugin for NetBeans. Let's go beyond the simple Toolbar Example and create a plugin which can auto-update itself. This code is based on the WakaTime plugin for NetBeans. Our example plugin will simply print a Hello World statement and update to new versions if available... just enough to get you started. Create a New Plugin Project Choose File -> New Project then NetBeans Modules -> Module as the project type. Name your project Choose a namespace or code name for your plugin Add a Java File Plugin Starting Point After creating the new Java Class file, make it extend ModuleInstall and wrap it with @OnShowing so it only runs after the GUI has loaded. @OnShowing public class MyPlugin extends ModuleInstall implements Runnable { } Press ALT + ENTER with your cursor over OnShowing then select Search Module Dependency for OnShowing to import the Window System API into the project. This will add a new dependency to your project as well as add the necessary import statements to the top of your file. Also do this for ModuleInstall. Sometimes NetBeans misses the org.openide.util dependency, so you might have to add that one manually. To do that, right click on MyPlugin then select Properties. Choose category Libraries then click Add.... Type org.openide.util then click OK. This will add the dependency to your project.xml file. Press ALT + ENTER on your MyPlugin class, then choose Implement all abstract methods. Now the run() method will execute after your plugin has loaded. Logging Let's make that println output to the NetBeans IDE log. First, setup the logger as an attribute of yourMyPlugin class. public static final Logger log = Logger.getLogger("MyPlugin"); Press ALT + ENTER to import java.util.logging.Logger. Replace println with log.info("MyPlugin has loaded.");. Updating Your Plugin Automatically Create a new Java file UpdateHandler.java inside your MyPlugin package. Replace the contents of this file with UpdateHandler.java. Search the module dependency and add any missing dependencies by pressing ALT + ENTER over each import statement. Add these lines to your manifest.mf file. OpenIDE-Module-Layer: org/myorg/myplugin/layer.xml OpenIDE-Module-Implementation-Version: 201501010101 Create a new XML document in your MyPlugin package. Add this code to your MyPlugin class inside the run() method. WindowManager.getDefault().invokeWhenUIReady(new Runnable () { @Override public void run() { UpdateHandler.checkAndHandleUpdates(); } }); Add these lines to your Bundle.properties file: Services/AutoupdateType/org_myorg_myplugin_update_center.instance=MyPlugin UpdateHandler.NewModules=false org_myorg_myplugin_update_center=https\://example.com/updates.xml Now every time NetBeans restarts and launches your plugin, it will check for updates by downloading updates.xml from example.com. Your updates.xml file tells NetBeans where to get the new NBM of your plugin. To create an NBM for publishing your plugin, right click on your MyPlugin project and select Create NBM. The NBM file is what you will publish to the NetBeans Plugin Portal. For an example of hosting updates.xml on GitHub, look at update.xml and corrospondingBundle.properties from the WakaTime NetBeans plugin.

First and foremost, it is important to define what AWS Activate is and what it is used for before we can take a deeper look. Exactly one year ago, Amazon created a program specifically designed for a particular group of customers that often times is in need of as much help as they can get (AKA startups). This program supports startups in their initial phase of building their businesses. This includes providing AWS credits, taking part in startup contests, and receiving benefits from third party solutions on the AWS cloud. Activate allows AWS partners that want to create a presence within the Activate community offer perks to member startups. Some of which include discounts and extended free tiers. Some startups that have attained high levels of success with AWS include Spotify, Pinterest, and Dropbox. With the big shots maintaining their places in startup stardom, Amazon has opened its doors to the next generation of innovators. As such, Amazon offers two different Activate packages. The Self-Starter package is comprised of a limited amount of each of the offerings listed above, whereas the Portfolio package includes some added bonuses along the lines of more high-profile and technical support as well as more in-depth training. On his blog AWS’ CTO, Werner Vogel, reiterated the importance of startups, “Startups will forever be a very important customer segment of AWS. They were among our first customers and along the way some amazing businesses have been built by these startups, many of which running for 100% on AWS.” “We’re excited to be a part of this global momentum in the startup ecosystem. The challenge now is to support and assist an increasing number of startups across the world.” The fun doesn’t stop there. In April of this year, AWS expanded the Activate package to offer much more than generalupport. This entailed sponsoring solution architects to take startups through step by step consultations in the fields of security, architecture and performance. Consequently, though Amazon’s professional services teams were established for customers, it was natural to have them take part in Activate. By nurturing new startups and making them rely heavily on the AWS cloud. As we can see today, companies that started with AWS 4 years ago are now worth billions of dollars. Airbnb and Dropbox, for example, now thoroughly enjoy the flexibility Amazon offers, as well as the fact that they no longer have to maintain cumbersome IT operations. Why not from the get-go? So the question is, if Amazon essentially built AWS on startups, why hasn’t Activate been around from the get-go, 6 years ago? AWS owes a great deal of its success to scalable startups that wanted and needed servers to run their businesses, yet didn’t have the initial capital to build their own data centers. No one really knows why Amazon did not provide startups back then with the kind of support they do today. However, as the market matured, it became clear that Amazon realized that an increasing number of startups could use their help. As a result, Amazon discovered that marketing their support services through Venture Capitalists and incubators around the world would include them as partners in this program and aid in marketing the service to startups of all kinds. “AWS Activate requires a special registration that allows startup customers with a valid AWS account to apply for either a self-starter package or a portfolio package. If a startup is a member of one of the accelerators, seed funds, or startup organizations that Amazon already works with, they may apply for the more exclusive AWS Activate Portfolio Package.” Learn More Incubators and Accelerators It was a natural step for Amazon to partner with accelerators all over the world with the Activate package. In addition to supporting startups, as mentioned above, these accelerators act as channels in the startup scene.At the first AWS re:Invent, Bezos jokes to his fellow investors, saying that eventually some of the investments will return to him because of how heavily the startup scene relies on Amazon. Activate and the approximately 150 accelerators across the world, including White Accel, Techstars, Appwest, and Battery Ventures, genuinely support and understand the values of the AWS service. They are happy to be able to use the Activate platform to help their startups flourish within the AWS clouds. 3rd Party Partners Aside from the accelerators, as an Amazon partner, you can enroll special offers to Activate members. For example, members that are part of the Self-Starter package may receive a 3 month free trial for Chef, whereas Portfolio members may receive a 6 month trial. Most of the partners will provide an extended free trial or credits via Activate. For instance, Trend Micro, one of Amazon’s biggest partners in the security domain, provides $2500 credit for Activate members in the Portfolio package. While there are not many partners on the list, the ones that are mentioned are very helpful and provide nice benefits for Activate members. Reviews of the program from both the partners’ and startups’ side showed that Activate is ideal for startups that have resource constraints. While members within the Self-Starter package are able to use the AWS Free Usage Tier, Portfolio members can receive anywhere from $1,000 to $15,000 in AWS Promotional Credit. The credit is maybe the most important value for these startups. Bearing in mind that Google also has their own line of packages and credit for new companies, it makes sense for AWS to start giving more life to these companies, above the free tier. Everyone has access to the free tier, these startups simply get more of it. Seems that there is no downside to participating. There is no obligation and the worst thing that can happen is that you will find that the services are great, and simply continue using them, which may result in you being locked-in to the point where you need to eventually pay. On the other hand, seems that the last announcement in April, which is actually “meet our architects”. Meaning the knowledge that Amazon’s architects share with startups in their consultation sessions help them get a better grasp on the ecosystem, as well as understand that more resource utilization is ultimately the next logical step for growth. All in all, although Amazon didn’t offer with this program 4 years ago, the AWS cloud was still the natural choice for startups. It included all of the benefits a startup can get using and online and on-demand infinite amount of resources. As a result, it is the clear choice for web scale startups. There are many reasons why Amazon only recently decided to offer free benefits to their prized potential customers. While it could have stemmed from competition from Microsoft and Google, or Amazon may want to simply show their support for their potential customers, demonstrating their cloud’s benefits at an early stage. Aside from that, Amazon understands and is built on companies with long term goals and possibilities. Therefore Amazon sees startups as a long term investment, which starts off with little risk.

Converters are an essential building block in XAML interfaces with one simple task: converting values of one type to another. Since they have a input, usually a view model property, and an output, it would be wonderful if we could somehow chain them to create a new converter that processes all internal converters. Luckily, this is quite simple to do, but we do need to create a new converter which will hold other converters and whose implementation will iterate over nested converters. Full code can be found over at Github repository here, only interesting parts will be highlighted in this blog post. Our combining converter class is also a converter itself, but it can contain other converters inside it: [ContentProperty("Converters")] public class ChainingConverter : IValueConverter { public Collection Converters { get; set; } } Converter functions are trivially implemented and iteratively go through the converters list and apply the converter on the previous value. public object Convert(object value, Type targetType, object parameter, CultureInfo culture) { foreach (var converter in Converters) { value = converter.Convert(value, targetType, parameter, culture); } return value; } ConvertBack is implemented in the same fashion. This allows us to create new converters in XAML with the following syntax: But what if we need to send parameters to some of the converters, how can we do that when the same parameter is used throughout the ChainingConverter implementation? To provide custom parameter for individual converters, we can create a wrapper converter around existing converter and specify parameter on that wrapper. Here is a skeleton for such wrapper converter, notice that the wrapper is also a converter: [ContentProperty("Converter")] public class ParameterizedConverterWrapper : DependencyObject, IValueConverter { // IValueConverter Converter dependency property // object Parameter dependency property // object DefaultReturnValue dependency property public object Convert(object value, Type targetType, object parameter, CultureInfo culture) { if (Converter != null) return Converter.Convert(value, targetType, Parameter ?? parameter, culture); return DefaultReturnValue; } } Converter wrappers allow us to create complex converters such as this one: The final converter should be self explanatory even though you probably haven’t seen these converters before. You can see that unlike other converters, the wrapper is a dependency object which allows us to use bindings on the Parameter property since it is in fact a dependency property. More complex converters should be created from ordinary converters whenever possible, especially when working with primitive types such as bool, string, enums and null values. What’s next? The last example looked like a small DSL embedded in XAML. We could create converters that simulate flow control or conditionals. We could even create converters that switch depending on the property before it, essentially coding logic inside such converters. Whether that is desirable is debatable, but it can be done. The full code with sample application can be found at the following Github repository: MassivePixel/wp-common.

This blog post is the third in a series that cover Azure Active Directory Single Sign-On (SSO) authentication in native mobile applications. Authenticating iOS app users with Azure Active Directory How to Best handle AAD access tokens in native mobile apps Using Azure SSO tokens for Multiple AAD Resources From Native Mobile Apps(this post) Sharing Azure SSO access tokens across multiple native mobile apps. Brief Start In an enterprise context, it is highly likely that you would have multiple web services that your native mobile app needs to consume. I had exactly this scenario, where one of my clients had asked if they could maintain the same token in the background in the mobile app to use it for accessing multiple web services. I spent some time digging through the documentation and conducting some experiments to confirm some points. Therefore, this post is to share my findings on accessing multiple Azure AD resources from native mobile apps using ADAL. In the previous two posts, we looked at implementing Azure AD SSO login on native mobile apps, then we looked at how to best maintain these access tokens. This post discusses how to use Azure AD SSO tokens to manage access to multiple AAD resources. Let’s assume that we have 2 web services sitting in Azure (ie WebApi1, and WebApi2), both of which are set to use Azure AD authentication. Then, we have the native mobile app, which needs access to both web services (WebApi1, and WebApi2). Let’s look at what we can and cannot do. Cannot Use the Same Azure AD Access-Token for Multiple Resources The first thing that comes to mind is to use the same access token for multiple Azure AD resources, and that is what the client asked about. However, this is not allowed. Azure AD issues a token for certain resource (which is mapped to an Azure AD app). When we call AcquireToken(), we need to provide a resourceID, only ONE resourceID. The result would have a token that can only be used for the supplied resource (id). There are ways where you could use the same token (as we will see later in this post), but it is not recommended as it complicates operations logging, authentication process tracing, etc. Therefore it is better to look at the other options provided by Azure and the ADAL library. Use the Refresh-Token to Acquire Tokens for Multiple Resources The ADAL library supports acquiring multiple access-Tokens for multiple resources using a refresh token. This means once a user is authenticated, the ADAL’s authentication context, would be able to generate an access-token to multiple resources without authenticating the user again. This was mentioned briefly by the MSDN documentation here. The refresh token issued by Azure AD can be used to access multiple resources. For example, if you have a client application that has permission to call two web APIs, the refresh token can be used to get an access token to the other web API as well. (MSDN documentation) public async Task RefreshTokens() { var tokenEntry = await tokensRepository.GetTokens(); var authorizationParameters = new AuthorizationParameters (_controller); var result = "Refreshed an existing Token"; bool hasARefreshToken = true; if (tokenEntry == null) { var localAuthResult = await _authContext.AcquireTokenAsync ( resourceId1, clientId, new Uri (redirectUrl), authorizationParameters, UserIdentifier.AnyUser, null); tokenEntry = new Tokens { WebApi1AccessToken = localAuthResult.AccessToken, RefreshToken = localAuthResult.RefreshToken, Email = localAuthResult.UserInfo.DisplayableId, ExpiresOn = localAuthResult.ExpiresOn }; hasARefreshToken = false; result = "Acquired a new Token"; } var refreshAuthResult = await _authContext.AcquireTokenByRefreshTokenAsync(tokenEntry.RefreshToken, clientId, resourceId2); tokenEntry.WebApi2AccessToken = refreshAuthResult.AccessToken; tokenEntry.RefreshToken = refreshAuthResult.RefreshToken; tokenEntry.ExpiresOn = refreshAuthResult.ExpiresOn; if (hasARefreshToken) { // this will only be called when we try refreshing the tokens (not when we are acquiring new tokens. refreshAuthResult = await _authContext.AcquireTokenByRefreshTokenAsync (refreshAuthResult.RefreshToken, clientId, resourceId1); tokenEntry.WebApi1AccessToken = refreshAuthResult.AccessToken; tokenEntry.RefreshToken = refreshAuthResult.RefreshToken; tokenEntry.ExpiresOn = refreshAuthResult.ExpiresOn; } await tokensRepository.InsertOrUpdateAsync (tokenEntry); return result; } As you can see from above, we check if we have an access-token from previous runs, and if we do, we refresh the access-tokens for both web services. Notice how the _authContext.AcquireTokenByRefreshTokenAsync() provides an overloading parameter that takes a resourceId. This enables us to get multiple access tokens for multiple resources without having to re-authenticate the user. The rest of the code is similar to what we have seen in the previous two posts. ADAL Library Can Produce New Tokens For Other Resources In the previous two posts, we looked at ADAL library and how it uses TokenCache. Although ADAL does not support persistent caching of tokens yet on mobile apps, it still uses the TokenCache for in-memory caching. This enables ADAL library to generate new access-tokens if the context (AuthenticationContext) still exists from previous authentications. Remember in the previous post we said it is recommended to keep a reference to the authentication-context? Here it comes in handy, as it enables us to generate new access-tokens for accessing multiple Azure AD resources. var localAuthResult = await _authContext.AcquireTokenAsync ( resourceId2, clientId, new Uri (redirectUrl), authorizationParameters, UserIdentifier.AnyUser, null); Calling AcquireToken() (even with no refresh-token) would give us a new access-token to webApi2. This is due to ADAL great goodness where it checks if we have a refresh-token in-memory (managed by ADAL), then it uses that to generate a new access-token for webApi2. An alternative The third alternative option is the simplest, but not necessarily the best. In this option, we could use the same access token to consume multiple Azure AD resources. To do this, we need to use the same Azure AD app ID when setting the web application’s authentication. This requires some understanding of how the Azure AD authentication happens on our web apps. If you refer to Taiseer Joudeh’s tutorial, which we mentioned before, you will see that in our web app, we need to tell the authentication framework what’s our Authority and the Audience (Azure AD app Id). If we set up both of our web apps, to use the same Audience (Azure AD app Id), meaning that we link them both into the same Azure AD application, then we could use the same access-token to use both web services. // linking our web app authentication to an Azure AD application private void ConfigureAuth(IAppBuilder app) { app.UseWindowsAzureActiveDirectoryBearerAuthentication( new WindowsAzureActiveDirectoryBearerAuthenticationOptions { Audience = ConfigurationManager.AppSettings["Audience"], Tenant = ConfigurationManager.AppSettings["Tenant"] }); } As we said before, this is very simple and requires less code, but could cause complications in terms of security logging and maintenance. At the end of the day, it depends on your context and what you are trying to achieve. Therefore, I thought it would be worth mentioning and I will leave the judgement for you on which option you choose. Conclusions We looked at how we could use Azure AD SSO with ADAL to access multiple resources from native mobile apps. As we saw, there are three main options, and the choice could be made based on the context of your app. I hope you find this useful and if you have any questions or you need help with some development that you are doing, then just get in touch. This blog post is the third in a series that cover Azure Active Directory Single Sign-On (SSO) authentication in native mobile applications.

both high availability (ha) and disaster recovery (dr) have been essential it topics. fundamentally ha is about fault tolerance relevant to the availability of an examined subject like application, database, vms, etc. while dr roots on the ability to resume operations in the aftermath of a catastrophic event. a fundamental difference of these two is that ha expects no down time and no data loss, while dr does. they are different issues and should be addressed separately. background for many it shops, either ha or dr has been a high risk and high cost item. both are essential to business continuity, while traditionally tough technical problems to solve with very significant and long-term commitments on resources. not only they are technically challenging, but a continual cost-cutting which has become an it standard practice in the past two decades makes purchasing hardware/software and constructing either ha or dr solution on premises further distant from it’s financial and technical realties. sense of urgency too often, the technical challenges and resource commitments overwhelm it and turn ha and dr into academic discussions, or symbolic items on a project checklist. at the same time, information is rapidly exploding as internet, mobility and social-network are becoming integral in our daily lives and businesses. there are progressively more data to process and store. for many businesses, the needs for ha and dr is urgent for better managing risks. and continual availability and on-demand recoverability of it are becoming increasingly critical. this is the reality, now the good news is that the recent introduction of cloud computing has fundamentally changed how an ha or dr solution can be implemented. microsoft azure is a vivid example of ha and dr solutions with significantly reduced the required financial commitment and involved technical complexities. the traditional approach by establishing redundancy and acquiring a physical dr site with long-term resources and financial commitments is now largely replaced with consumable services which can be configured in minutes by mouse-clicking and with a manageable cost structure based on usage. ha and dr have become it solutions which are financially realistic and technically feasible for businesses in all sizes. ha, redundancy, and microsoft azure lrs ha is to eliminate a single point of failure of an examined component, an application for example. it denotes a strategy to employ redundancy such that a target application can and will continue being available without downtime while experiencing a failure of hosting hardware or software. there are various and well-developed ha solutions like a hyper-v host cluster using redundant hardware to eliminate a single point of failure of hosting os or hardware, and an application cluster for eliminating a single point of failure by running the application in multiple vm instances with a synchronous state. although ha implementations may vary, the fundamental principle nevertheless remains the same. ha expects neither downtime nor data loss while experiencing an outage of a target hardware or software. ha has become dramatically simple in microsoft azure. basically, all data written to disk in microsoft azure are kept at least in the so-called lrs, locally redundant storage. lrs replicates a transaction synchronously to three different storage nodes across fault domains and upgrade domains within the same region for durability. in layman’s terms, microsoft azure by default maintains at least three copies of user data to achieve ha. dr, replication, and microsoft azure grs dr is about having a plan and backups in place to resume operations in the aftermath of a catastrophic event. unplanned outage is assumed in a dr scenario, therefore some data loss is also expected. notice that ha and dr are different business problems and addressed differently. while both ha and dr are based on applying redundancy, i.e. a source and replicas, or multiple identical nodes of an examines component like application instance, databases, or vms, there are however differences between the two. a dr solution generally employs replicas or backups, are implemented with asynchronous processes, and expects an outage of a source and with some data loss in transit while the outage occurs. while ha requires a logical representation with a real-time integrity using synchronous processes across all participating nodes, expects neither downtime nor data loss while experiencing an outage of a participating node. for a critical workload, one approach of dr is to establish geo-replication to address an outage of an entire geographic area caused by a natural disaster, for example. the concern is that a catastrophic event may impact an entire geographic area causing a datacenter where a mission critical application is being hosted becomes unavailable for an extended period of time. in microsoft azure, geo redundant storage or grs is the default and an optional setting, as shown above, while configuring a storage account. grs will queue a transaction committed to lrs as an asynchronous replication to a secondary region, a few hundreds miles away from the primary region where a storage account is originated. at the secondary region, data is also stored in lrs, i.e. made durable by replicating it to three storage nodes. specifically, a microsoft azure storage account configured with grs essentially maintains three replicas locally for high availability, and replicates the content and maintains three replicas at a secondary datacenter a few hundreds miles away for dr. so all are six copies, three locally and three remotely. all these are configured by one, yes one mouse click from a dropdown list while creating a storage account. the above is a conceptual model illustrated a data flow of grs. grs replication has little performance impact on an application since application data are committed to lrs in real-time while replication to grs is queued, i.e. asynchronously. a write to lrs is synchronous and in real-time, once committed, the changes are expected within 15 minutes to be asynchronously replicated to the secondary site. for a ra-grs storage account, in addition to one primary endpoint for read/write operations as it is in a grs, there is also one secondary endpoint as read only becomes available as shown below. the cost implications of grs or ra-grs include the additional storage and the transmission costs for egress traffic, as applicable, of the secondary datacenter. ingress traffic is free . and microsoft azure storage sla offers 99.9% availability and a cost calculator is also available. microsoft azure recovery services so far, much is about backing up or replicating data. to successfully restore, a dr plan must be put in place and ensure its availability upon a dr scenario in progress. either placing a dr plan at a primary site where the source is or a secondary site where a replica stays has some issues and concerns. keeping a dr plan at the source site where all the resources are in place and on-the-job trainings seems logical. or does it? dr is assuming a catastrophic event over an extended geographic areas where the source site is experiencing an outage. in such case, keeping a dr plan in the source site defeats the purpose. maintaining a dr plan at the secondary site is the choice then. in a dr scenario, a recovery site is to be brought on line within a expected period of time according to a dr plan, and having the dr plan right there and then at a recovery site makes all the sense. or does it? this decision introduces a number of requirements including the physical readiness, the timeliness, and the financial implications on securing and maintaining a dr plan at a remote physical facility. for a vmm server running on system center 2012 sp1 or later, an idea, reliable and straightforward way is to use azure recovery services to maintain a dr plan as shown below. and for any backup needs, using cloud as a backup site makes backing up and restoring data an anytime anywhere operation. azure site recovery vault this service essentially acts as the director of a dr process. it orchestrates and manages the protection and failover of vms in clouds managed by virtual machine manager 2012 sp1 or later. a noticeable advantage is the ability to test a recovery configuration, exercise a proactive failover and recovery, and automate recovery in the event of a site outage. the sla of site recovery services is 99.9% availability to ensure a configured dr plan is always in place with expected updates. this is a dr solution that it can implement, simulate, verify, bring online and be absolutely confident with the readiness. azure backup vault this is a reliable, scalable and inexpensive data protection solution with zero capital investment and extremely low operational expense. like other secure communication with microsoft azure, you will first upload a public certificate to microsoft azure. then download the backup agent to register a target server with the backup vault. then select what to be backed up. both microsoft azure backup sla (99.9% availability) and cost calculator are available for better assessing the solution. closing thoughts form an application’s view, ha is an on-going event while dr is an anticipation. ha and dr are different business problems and should be addressed differently. nevertheless, microsoft azure provides a single platform to gracefully address ha with lrs, dr with grs, and dr orchestration with recovery services, and all with published sla s and a predictable cost structure . going forward, it pros can now include ha and dr as a reliable, scalable and relatively inexpensive proposition by employing microsoft azure as a solution platform. call to action register at microsoft virtual academy, http://aka.ms/mva1 , and train yourself on microsoft azure by taking the track of courses. go to http://aka.ms/azure200 and acquire a free trial subscription and assess microsoft azure for ha and dr solutions. review my recommended content at http://aka.ms/recommended .

Use IntelliJ Community Edition, Tomcat and Tomcat Maven Plugin.

[This article was written by Yaron Parasol.] Docker containers were created to help enable the fast, and reliable deployment of application components or tiers, by creating a container that holds a self-contained ready to deploy parts of applications, with the middleware and the app business logic needed to run them successfully. For example, a Spring application within a Tomcat container. By design, Docker is purposely an isolated self-contained part of the application, typically one tier or even one node in a tier. However, an application is typically multi-tier in its architecture and that means you have tiers with dependencies between them, where the nature of the dependencies can be anything from network connections and remote API invocations, to exchange of messages between application tiers. And hence an app is a set of different containers with specific configurations. This is why you need a way to glue the pieces of your app together. While, Docker has a basic solution for connecting containers using a Docker bridge, this solution is not always the preferred one, especially when deploying the container across different hosts and you need to take care of real network settings. Docker orchestration with TOSCA + Cloudify. Check it out. Go So, what role does the orchestrator play? The orchestrator will take care of two things: The timing of container creation - as containers need to be created by order of dependencies and Container configuration in order to allow containers to communicate with one another - and for that the orchestrator needs to pass runtime properties between containers. As a side note here: With Docker you need a special tweak here, as you typically don’t touch config files inside a container, you keep the container intact, so there is an interesting workaround for cases that this is required. One method to do this is by using a YAML-based orchestration plan to orchestrate the deployment of apps and post-deployment automation processes, which is the approach Cloudify employs. Based on TOSCA (topology and orchestration standard of cloud apps), this orchestration plan describes the components and their lifecycle, and the relationships between components, especially when it comes to complex topologies. This includes, what’s connected to what, what’s hosted on what, and other such considerations. TOSCA is able to describe the infrastructure, as well as, the middleware tier, and app layers on top of these. Cloudify basically takes this TOSCA orchestration plan (dubbed blueprints in Cloudify speak) and materializes these using workflows that traverse the graph of components, or this plan of components and issues commands to agents. These then create the app components and glue them together. The agents use extensions called plugins that are adaptors between the Cloudify configuration and the various infrastructure as a service (IaaS) and automation tools’ APIs. In our case, we created a plugin to interface with the Docker API. Introducing the Docker Cloudify Plugin The Cloudify-Docker plugin is quite straightforward, it installs the Docker API endpoint/server on the machine and then uses the Docker-Py binding to create, configure, and remove containers. TOSCA lifecycle events are: Create - installation of the app components Configure - configuration of the component Start - startup/running the component There is also stop & delete - for shutdown and removal We started by using the create - to create the container, we did not implement configure at the beginning, and start to run the application. But then we realized that for containers with dependencies we need to have runtime properties, such as IP import of the counterpart container in order to create the container for example. When we create an app server container, we need the port and IP of the database container. So, we pushed the creation of the container to the configure event, and used a TOSCA relationship pre-configure hook, to get the dependent container’s info at runtime. The way to expose the runtime info to the container with the dependencies is by setting them as environment variables. 01.interfaces: 02. cloudify.interfaces.lifecycle: 03. configure: 04. implementation: docker.docker_plugin.tasks.configure 05. inputs: 06. container_config: 07. command: mongod--rest--httpinterface --smallfiles 08. image: dockerfile/mongodb 09. start: 10. implementation: docker.docker_plugin.tasks.run 11. inputs: 12. container_start: 13. port_bindings: 14. 27017: 27017 15. 28017: 28017 Nodecellar Example I’d like to explain how this works by using our Nodecellar app as an example. The Nodecellar app is composed of two hosts that, in this case, Cloudify didn’t create but just SSHed into and then installed agents on. On one we have the MongoD container, with a MongoD process. On the other we have the Nodecellar container with NodeJS and the Nodecellar app within it. The Nodecellar container needs a connection to the MongoD container to run the app queries when the app starts. Ultimately, an orchestrator should not be limited to software deployment, the whole idea behind Docker Is to allow for agility, so we’d also like to use Docker in situations of auto-scale out and auto-heal, CD. In our next post we’ll show exactly that - how Cloudify can be used with Docker for post-deployment scenarios.

I’m a huge Vaadin fan and I’ve created a Github workshop I can demo at conferences. A common issue with such kind of workshops is that attendees have to prepare their workstations in advance… and there’s always a significant part of them that comes with not everything ready. At this point, two options are available to the speaker: either wait for each of the attendee to finish the preparation – too bad for the people who took the time at home to do that, or start anyway – and lose the not-ready part. Given the current buzz around Docker, I thought that could be a very good way to make the workshop preparation quicker – only one step, and hasslefree – no problem regarding the quirks of your operation system. The required steps I ask the attendees are the following: Install Git Install Java, Maven and Tomcat Clone the git repo Build the project (to prepare the Maven repository) Deploy the built webapp Start Tomcat These should directly be automated into Docker. As I wasted much time getting this to work, here’s the tale of my journey in achieving this (be warned, it’s quite long). If you’ve got similar use-cases, I hope it will be useful in you getting things done faster. Starting with Docker The first step was to get to know the basics about Docker. Fortunately, I had the chance to attend a Docker workshop by David Gageot at Duchess Swiss. This included both Docker installation and basics of Dockerfile. I assume readers have likewise a basic understanding of Docker. For those who don’t, I guess browsing the Docker’s official documentation is a nice idea: Installation Dockerfile reference Building my first Dockerfile The Docker image can be built with the following command ran into the directory of the Dockerfile: $ docker build -t vaadinworkshop . The first issues one can encounter when playing with Docker the first time, is to get the following error message: Get http:///var/run/docker.sock/v1.14/containers/json: dial unix /var/run/docker.sock: no such file or directory The reason is because one didn’t export the required environment variables displayed by the boot2docker information message. If you lost the exact data, no worry, just use the shellinit boot2docker parameter: $ boot2docker shellinit Writing /Users/i303869/.docker/boot2docker-vm/ca.pem: Writing /Users/i303869/.docker/boot2docker-vm/cert.pem: Writing /Users/i303869/.docker/boot2docker-vm/key.pem: export DOCKER_HOST=tcp://192.168.59.103:2376 export DOCKER_CERT_PATH=/Users/i303869/.docker/boot2docker-vm Copy-paste the export lines above will solve the issue. These can also be set in one’s .bashrc script as it seems these values seldom change. Next in line is the following error: Get http://192.168.59.103:2376/v1.14/containers/json: malformed HTTP response "x15x03x01x00x02x02" This error message seems to be because of a mismatch between versions of the client and the server. It seems it is because of a bug on Mac OSX when upgrading. For a long term solution, reinstall Docker from scratch; for a quick fix, use the --tls flag with the docker command. As it is quite cumbersome to type it everything, one can alias it: $ alias docker="docker --tls" My last mistake when building the image comes from building the Dockerfile from a not empty directory. Docker sends every file it finds in the directory of the Dockerfile to the Docker container for build: $ docker --tls build -t vaadinworkshop . Sending build context to Docker daemon Too many kB Fix: do not try this at home and start from a directory container the Dockerfile only. Starting from scratch Dockerfiles describe images – images are built as a layered list of instructions. Docker images are designed around single inheritance: one image has to be set a single parent. An image requiring no parent starts from scratch, but Docker provides 4 base official distributions: busybox, debian, ubuntu and centos (operating systems are generally a good start). Whatever you want to achieve, it is necessary to choose the right parent. Given the requirements I set for myself (Java, Maven, Tomcat and Git), I tried to find the right starting image. Many Dockerfiles are already available online on the Docker hub. The browsing app is quite good, but to be really honest, the search can really be improved. My intention was to use the image that matched the most of my requirements, then fill the gap. I could find no image providing Git, but I thought the dgageot/maven Dockerfile would be a nice starting point. The problem is that the base image is a busybox and provides no installer out-of-the-box (apt-get, yum, whatever). For this reason, David uses a lot of curl to get Java 8 and Maven in his Dockerfiles. I foolishly thought I could use a different flavor of busybox that provides the opkg installer. After a while, I accumulated many problems, resolving one heading to another. In the end, I finally decided to use the OS I was most comfortable with and to install everything myself: FROM ubuntu:utopic Scripting Java installation Installing git, maven and tomcat packages is very straightforward (if you don’t forget to use the non-interactive options) with RUN and apt-get: RUN apt-get update && \ apt-get install -y --force-yes git maven tomcat8 Java doesn’t fall into this nice pattern, as Oracle wants you to accept the license. Nice people did however publish it to a third-party repo. Steps are the following: Add the needed package repository Configure the system to automatically accept the license Configure the system to add un-certified packages Update the list of repositories At last, install the package Also add a package for Java 8 system configuration. RUN echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list && \ echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && \ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 RUN apt-get update && \ apt-get install -y --force-yes oracle-java8-installer oracle-java8-set-default Building the sources Getting the workshop’s sources and building them is quite straightforward with the following instructions: RUN git clone https://github.com/nfrankel/vaadin7-workshop.git WORKDIR /vaadin7-workshop RUN mvn package The drawback of this approach is that Maven will start from a fresh repository, and thus download the Internet the first time it is launched. At first, I wanted to mount a volume from the host to the container to share the ~/.m2/repository folder to avoid this, but I noticed this could only be done at runtime through the -v option as the VOLUME instruction cannot point to a host directory. Starting the image The simplest command to start the created Docker image is the following: $ docker run -p 8080:8080 Do not forget the port forwarding from the container to the host, 8080 for the standard HTTP port. Also, note that it’s not necessary to run the container as a daemon (with the -d option). The added value of that is that the standard output of the CMD (see below) will be redirected to the host. When running as a daemon and wanting to check the logs, one has to execute bash in the container, which requires a sequence of cumbersome manipulations. Configuring and launching Tomcat Tomcat can be launched when starting the container by just adding the following instruction to the Dockerfile: CMD ["catalina.sh", "run"] However, trying to start the container at this point will result in the following error: Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/common/classes], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/common], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/server/classes], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/server], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/shared/classes], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.ClassLoaderFactory validateFile WARNING: Problem with directory [/usr/share/tomcat8/shared], exists: [false], isDirectory: [false], canRead: [false] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina initDirs SEVERE: Cannot find specified temporary folder at /usr/share/tomcat8/temp Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina load WARNING: Unable to load server configuration from [/usr/share/tomcat8/conf/server.xml] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina initDirs SEVERE: Cannot find specified temporary folder at /usr/share/tomcat8/temp Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina load WARNING: Unable to load server configuration from [/usr/share/tomcat8/conf/server.xml] Nov 15, 2014 9:24:18 PM org.apache.catalina.startup.Catalina start SEVERE: Cannot start server. Server instance is not configured. I have no idea why, but it seems Tomcat 8 on Ubuntu is not configured in any meaningful way. Everything is available but we need some symbolic links here and there as well as creating the temp directory. This translates into the following instruction in the Dockerfile: RUN ln -s /var/lib/tomcat8/common $CATALINA_HOME/common && \ ln -s /var/lib/tomcat8/server $CATALINA_HOME/server && \ ln -s /var/lib/tomcat8/shared $CATALINA_HOME/shared && \ ln -s /etc/tomcat8 $CATALINA_HOME/conf && \ mkdir $CATALINA_HOME/temp The final trick is to connect the exploded webapp folder created by Maven to Tomcat’s webapps folder, which it looks for deployments: RUN mkdir $CATALINA_HOME/webapps && \ ln -s /vaadin7-workshop/target/workshop-7.2-1.0-SNAPSHOT/ $CATALINA_HOME/webapps/vaadinworkshop At this point, the Holy Grail is not far away, you just have to browse the URL… if only we knew what the IP was. Since running on Mac, there’s an additional VM beside the host and the container that’s involved. To get this IP, type: $ boot2docker ip The VM's Host only interface IP address is: 192.168.59.103 Now, browsing http://192.168.59.103:8080/vaadinworkshop/ will bring us to the familiar workshop screen: Developing from there Everything works fine but didn’t we just forget about one important thing, like how workshop attendees are supposed to work on the sources? Easy enough, just mount the volume when starting the container: docker run -v /Users//vaadin7-workshop:/vaadin7-workshop -p 8080:8080 vaadinworkshop Note that the host volume must be part of /Users and if on OSX, it must use boot2docker v. 1.3+. Unfortunately, it seems now is the showstopper, as mounting an empty directory from the host to the container will not make the container’s directory available from the host. On the contrary, it will empty the container’s directory given that the host’s directory doesn’t exist… It seems there’s an issue in Docker on Mac. The installation of JHipster runs into the same problem, and proposes to use the Samba Docker folder sharing project. I’m afraid I was too lazy to go further at this point. However, this taught me much about Docker, its usages and use-cases (as well as OSX integration limitations). For those who are interested, you’ll find below the Docker file. Happy Docker! FROM ubuntu:utopic MAINTAINER Nicolas Frankel # Config to get to install Java 8 w/o interaction RUN echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list && echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections && apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 RUN apt-get update && apt-get install -y --force-yes git oracle-java8-installer oracle-java8-set-default maven tomcat8 RUN git clone https://github.com/nfrankel/vaadin7-workshop.git WORKDIR /vaadin7-workshop RUN git checkout v7.2-1 RUN mvn package ENV JAVA_HOME /usr/lib/jvm/java-8-oracle ENV CATALINA_HOME /usr/share/tomcat8 ENV PATH $PATH:$CATALINA_HOME/bin # Configure Tomcat 8 directories RUN ln -s /var/lib/tomcat8/common $CATALINA_HOME/common && ln -s /var/lib/tomcat8/server $CATALINA_HOME/server && ln -s /var/lib/tomcat8/shared $CATALINA_HOME/shared && ln -s /etc/tomcat8 $CATALINA_HOME/conf && mkdir $CATALINA_HOME/temp && mkdir $CATALINA_HOME/webapps && ln -s /vaadin7-workshop/target/workshop-7.2-1.0-SNAPSHOT/ $CATALINA_HOME/webapps/vaadinworkshop VOLUME ["/vaadin7-workshop"] CMD ["catalina.sh", "run"] # docker build -t vaadinworkshop . # docker run -v ~/vaadin7-workshop training/webapp -p 8080:8080 vaadinworkshop

I was involved in a project recently where we needed to capture the http session id for a websocket request - the reason was to determine the number of websocket sessions utilizing the same underlying http session The way to do this is based on a sample utilizing the new spring-session module and is described here. The trick to capturing the http session id is in understanding that before a websocket connection is established between the browser and the server, there is a handshake phase negotiated over http and the session id is passed to the server during this handshake phase. Spring Websocket support provides a nice way to register a HandShakeInterceptor, which can be used to capture the http session id and set this in the sub-protocol(typically STOMP) headers. First, this is the way to capture the session id and set it to a header: public class HttpSessionIdHandshakeInterceptor implements HandshakeInterceptor { @Override public boolean beforeHandshake(ServerHttpRequest request, ServerHttpResponse response, WebSocketHandler wsHandler, Map attributes) throws Exception { if (request instanceof ServletServerHttpRequest) { ServletServerHttpRequest servletRequest = (ServletServerHttpRequest) request; HttpSession session = servletRequest.getServletRequest().getSession(false); if (session != null) { attributes.put("HTTPSESSIONID", session.getId()); } } return true; } public void afterHandshake(ServerHttpRequest request, ServerHttpResponse response, WebSocketHandler wsHandler, Exception ex) { } } And to register this HandshakeInterceptor with Spring Websocket support: @Configuration @EnableWebSocketMessageBroker public class WebSocketDefaultConfig extends AbstractWebSocketMessageBrokerConfigurer { @Override public void configureMessageBroker(MessageBrokerRegistry config) { config.enableSimpleBroker("/topic/", "/queue/"); config.setApplicationDestinationPrefixes("/app"); } @Override public void registerStompEndpoints(StompEndpointRegistry registry) { registry.addEndpoint("/chat").withSockJS().setInterceptors(httpSessionIdHandshakeInterceptor()); } @Bean public HttpSessionIdHandshakeInterceptor httpSessionIdHandshakeInterceptor() { return new HttpSessionIdHandshakeInterceptor(); } } Now that the session id is a part of the STOMP headers, this can be grabbed as a STOMP header, the following is a sample where it is being grabbed when subscriptions are registered to the server: @Component public class StompSubscribeEventListener implements ApplicationListener { private static final Logger logger = LoggerFactory.getLogger(StompSubscribeEventListener.class); @Override public void onApplicationEvent(SessionSubscribeEvent sessionSubscribeEvent) { StompHeaderAccessor headerAccessor = StompHeaderAccessor.wrap(sessionSubscribeEvent.getMessage()); logger.info(headerAccessor.getSessionAttributes().get("HTTPSESSIONID").toString()); } } or it can be grabbed from a controller method handling websocket messages as a MessageHeaders parameter: @MessageMapping("/chats/{chatRoomId}") public void handleChat(@Payload ChatMessage message, @DestinationVariable("chatRoomId") String chatRoomId, MessageHeaders messageHeaders, Principal user) { logger.info(messageHeaders.toString()); this.simpMessagingTemplate.convertAndSend("/topic/chats." + chatRoomId, "[" + getTimestamp() + "]:" + user.getName() + ":" + message.getMessage()); }

The last couple of weeks I've been playing around with docker and kubernetes. If you are not familiar with kubernetes let's just say for now that its an open source container cluster management implementation, which I find really really awesome. One of the first things I wanted to try out was running an Apache ZooKeeper ensemble inside kubernetes and I thought that it would be nice to share the experience. For my experiments I used Docker v. 1.3.0 and Openshift V3, which I built from source and includes Kubernetes. ZooKeeper on Docker Managing a ZooKeeper ensemble is definitely not a trivial task. You usually need to configure an odd number of servers and all of the servers need to be aware of each other. This is a PITA on its own, but it gets even more painful when you are working with something as static as docker images. The main difficulty could be expressed as: "How can you create multiple containers out of the same image and have them point to each other?" One approach would be to use docker volumes and provide the configuration externally. This would mean that you have created the configuration for each container, stored it somewhere in the docker host and then pass the configuration to each container as a volume at creation time. I've never tried that myself, I can't tell if its a good or bad practice, I can see some benefits, but I can also see that this is something I am not really excited about. It could look like this: docker run -p 2181:2181 -v /path/to/my/conf:/opt/zookeeper/conf my/zookeeper An other approach would be to pass all the required information as environment variables to the container at creation time and then create a wrapper script which will read the environment variables, modify the configuration files accordingly, launch zookeeper. This is definitely easier to use, but its not that flexible to perform other types of tuning without rebuilding the image itself. Last but not least one could combine the two approaches into one and do something like: Make it possible to provide the base configuration externally using volumes. Use env and scripting to just configure the ensemble. There are plenty of images out there that take one or the other approach. I am more fond of the environment variables approach and since I needed something that would follow some of the kubernetes conventions in terms of naming, I decided to hack an image of my own using the env variables way. Creating a custom image for ZooKeeper I will just focus on the configuration that is required for the ensemble. In order to configure a ZooKeeper ensemble, for each server one has to assign a numeric id and then add in its configuration an entry per zookeeper server, that contains the ip of the server, the peer port of the server and the election port. The server id is added in a file called myid under the dataDir. The rest of the configuration looks like: server.1=server1.example.com:2888:3888 server.2=server2.example.com:2888:3888 server.3=server3.example.com:2888:3888 ... server.current=[bind address]:[peer binding port]:[election biding port]Note that if the server id is X the server.X entry needs to contain the bind ip and ports and not the connection ip and ports. So what we actually need to pass to the container as environment variables are the following: The server id. For each server in the ensemble: The hostname or ip The peer port The election port If these are set, then the script that updates the configuration could look like: if [ ! -z "$SERVER_ID" ]; then echo "$SERVER_ID" > /opt/zookeeper/data/myid #Find the servers exposed in env. for i in `echo {1..15}`;do HOST=`envValue ZK_PEER_${i}_SERVICE_HOST` PEER=`envValue ZK_PEER_${i}_SERVICE_PORT` ELECTION=`envValue ZK_ELECTION_${i}_SERVICE_PORT` if [ "$SERVER_ID" = "$i" ];then echo "server.$i=0.0.0.0:2888:3888" >> conf/zoo.cfg elif [ -z "$HOST" ] || [ -z "$PEER" ] || [ -z "$ELECTION" ] ; then #if a server is not fully defined stop the loop here. break else echo "server.$i=$HOST:$PEER:$ELECTION" >> conf/zoo.cfg fi done fi For simplicity the function that read the keys and values from env are excluded. The complete image and helping scripts to launch zookeeper ensembles of variables size can be found in the fabric8io repository. ZooKeeper on Kubernetes The docker image above, can be used directly with docker, provided that you take care of the environment variables. Now I am going to describe how this image can be used with kubernetes. But first a little rambling... What I really like about using kubernetes with ZooKeeper, is that kubernetes will recreate the container, if it dies or the health check fails. For ZooKeeper this also means that if a container that hosts an ensemble server dies, it will get replaced by a new one. This guarantees that there will be constantly a quorum of ZooKeeper servers. I also like that you don't need to worry about the connection string that the clients will use, if containers come and go. You can use kubernetes services to load balance across all the available servers and you can even expose that outside of kubernetes. Creating a Kubernetes confing for ZooKeeper I'll try to explain how you can create 3 ZooKeeper Server Ensemble in Kubernetes. What we need is 3 docker containers all running ZooKeeper with the right environment variables: { "image": "fabric8/zookeeper", "name": "zookeeper-server-1", "env": [ { "name": "ZK_SERVER_ID", "value": "1" } ], "ports": [ { "name": "zookeeper-client-port", "containerPort": 2181, "protocol": "TCP" }, { "name": "zookeeper-peer-port", "containerPort": 2888, "protocol": "TCP" }, { "name": "zookeeper-election-port", "containerPort": 3888, "protocol": "TCP" } ] } The env needs to specify all the parameters discussed previously. So we need to add along with the ZK_SERVER_ID, the following: ZK_PEER_1_SERVICE_HOST ZK_PEER_1_SERVICE_PORT ZK_ELECTION_1_SERVICE_PORT ZK_PEER_2_SERVICE_HOST ZK_PEER_2_SERVICE_PORT ZK_ELECTION_2_SERVICE_PORT ZK_PEER_3_SERVICE_HOST ZK_PEER_3_SERVICE_PORT ZK_ELECTION_3_SERVICE_PORT An alternative approach could be instead of adding all these manual configuration, to expose peer and election as kubernetes services. I tend to favor the later approach as it can make things simpler when working with multiple hosts. It's also a nice exercise for learning kubernetes. So how do we configure those services? To configure them we need to know: the name of the port the kubernetes pod the provide the service The name of the port is already defined in the previous snippet. So we just need to find out how to select the pod. For this use case, it make sense to have a different pod for each zookeeper server container. So we just need to have a label for each pod, the designates that its a zookeeper server pod and also a label that designates the zookeeper server id. "labels": { "name": "zookeeper-pod", "server": 1 } Something like the above could work. Now we are ready to define the service. I will just show how we can expose the peer port of server with id 1, as a service. The rest can be done in a similar fashion: { "apiVersion": "v1beta1", "creationTimestamp": null, "id": "zk-peer-1", "kind": "Service", "port": 2888, "containerPort": "zookeeper-peer-port", "selector": { "name": "zookeeper-pod", "server": 1 } } The basic idea is that in the service definition, you create a selector which can be used to query/filter pods. Then you define the name of the port to expose and this is pretty much it. Just to clarify, we need a service definition just like the one above per zookeeper server container. And of course we need to do the same for the election port. Finally, we can define an other kind of service, for the client connection port. This time we are not going to specify the sever id, in the selector, which means that all 3 servers will be selected. In this case kubernetes will load balance across all ZooKeeper servers. Since ZooKeeper provides a single system image (it doesn't matter on which server you are connected) then this is pretty handy. { "apiVersion": "v1beta1", "creationTimestamp": null, "id": "zk-client", "kind": "Service", "port": 2181, "createExternalLoadBalancer": "true", "containerPort": "zookeeper-client-port", "selector": { "name": "zookeeper-pod" } } The basic idea is that in the service definition, you create a selector which can be used to query/filter pods. Then you define the name of the port to expose and this is pretty much it. Just to clarify, we need a service definition just like the one above per zookeeper server container. And of course we need to do the same for the election port. Finally, we can define an other kind of service, for the client connection port. This time we are not going to specify the sever id, in the selector, which means that all 3 servers will be selected. In this case kubernetes will load balance across all ZooKeeper servers. Since ZooKeeper provides a single system image (it doesn't matter on which server you are connected) then this is pretty handy. { "apiVersion": "v1beta1", "creationTimestamp": null, "id": "zk-client", "kind": "Service", "port": 2181, "createExternalLoadBalancer": "true", "containerPort": "zookeeper-client-port", "selector": { "name": "zookeeper-pod" } } I hope you found it useful. There is definitely room for improvement so feel free to leave comments.

NetBeans 8.0 introduces several new Java hints. Although there are a large number of these new hints related to Java Persistence API, I focus on five new hints in the Performance category. The five new "Performance Hints" introduced with NetBeans 8.0 are: Boxing of already boxed value Redundant String.toString() Replace StringBuffer/StringBuilder by String Unnecessary temporary during conversion from String Unnecessary temporary during conversion to String Each of these five performance-related Java hints are illustrated in this post with screen snapshots taken from NetBeans 8.0 with code that demonstrates these hints. There are two screen snapshots for each hint, one each showing the text displayed when the cursor hovers over the line of code marked with yellow underlining and one each showing the suggested course of action to be applied to address that hint (shown when clicking on the yellow light bulb to the left of the flagged line). Some of the captured screen snapshots include examples of code that avoid the hint. Boxing of Already Boxed Value Redundant String.toString() Replace StringBuffer/StringBuilder by String Unnecessary Temporary During Conversion from String Unnecessary Temporary During Conversion to String Unless I've done something incorrectly, there appears to be a minor bug with this hint in that it reports "Unnecessary temporary when converting from String" when, in this case, it should really be "Unnecessary temporary when converting to String". This is not a big deal as the condition is flagged and the action to fix it seems appropriate. Conclusion The five performance-related hints introduced by NetBeans 8.0 and illustrated here can help Java developers avoid unnecessary object instantiations and other unnecessary runtime cost. Although the benefit of this optimization as shown in my simple examples is almost negligible, it could lead to much greater savings when used in code with loops that perform these same unnecessary instantiations thousands of times. Even without consideration of the performance benefit, these hints help to remind Java developers and teach developers new to Java about the most appropriate mechanisms for acquiring instances and primitive values.

One of my favourite functions in Neo4j’s cypher query language is COLLECT which allows us to group items into an array for later consumption. However, I’ve noticed that people sometimes have trouble working out how to collect multiple items with COLLECT and struggle to find a way to do so. Consider the following data set: create (p:Person {name: "Mark"}) create (e1:Event {name: "Event1", timestamp: 1234}) create (e2:Event {name: "Event2", timestamp: 4567}) create (p)-[:EVENT]->(e1) create (p)-[:EVENT]->(e2) If we wanted to return each person along with a collection of the event names they’d participated in we could write the following: $ MATCH (p:Person)-[:EVENT]->(e) > RETURN p, COLLECT(e.name); +--------------------------------------------+ | p | COLLECT(e.name) | +--------------------------------------------+ | Node[0]{name:"Mark"} | ["Event1","Event2"] | +--------------------------------------------+ 1 row That works nicely, but what about if we want to collect the event name and the timestamp but don’t want to return the entire event node? An approach I’ve seen a few people try during workshops is the following: MATCH (p:Person)-[:EVENT]->(e) RETURN p, COLLECT(e.name, e.timestamp) Unfortunately this doesn’t compile: SyntaxException: Too many parameters for function 'collect' (line 2, column 11) "RETURN p, COLLECT(e.name, e.timestamp)" ^ As the error message suggests, the COLLECT function only takes one argument so we need to find another way to solve our problem. One way is to put the two values into a literal array which will result in an array of arrays as our return result: $ MATCH (p:Person)-[:EVENT]->(e) > RETURN p, COLLECT([e.name, e.timestamp]); +----------------------------------------------------------+ | p | COLLECT([e.name, e.timestamp]) | +----------------------------------------------------------+ | Node[0]{name:"Mark"} | [["Event1",1234],["Event2",4567]] | +----------------------------------------------------------+ 1 row The annoying thing about this approach is that as you add more items you’ll forget in which position you’ve put each bit of data so I think a preferable approach is to collect a map of items instead: $ MATCH (p:Person)-[:EVENT]->(e) > RETURN p, COLLECT({eventName: e.name, eventTimestamp: e.timestamp}); +--------------------------------------------------------------------------------------------------------------------------+ | p | COLLECT({eventName: e.name, eventTimestamp: e.timestamp}) | +--------------------------------------------------------------------------------------------------------------------------+ | Node[0]{name:"Mark"} | [{eventName -> "Event1", eventTimestamp -> 1234},{eventName -> "Event2", eventTimestamp -> 4567}] | +--------------------------------------------------------------------------------------------------------------------------+ 1 row During the Clojure Neo4j Hackathon that we ran earlier this week this proved to be a particularly pleasing approach as we could easily destructure the collection of maps in our Clojure code.

It is possible to disable HTTP access on S3 bucket, limiting S3 traffic to only HTTPS requests. The documentation is scattered around the Amazon AWS documentation, but the solution is actually straightforward. All you need to do to block HTTP traffic on an S3 bucket is add a Condition in your bucket's policy. AWS supports a global condition for verifying SSL. So you can add a condition like this: "Condition": { "Bool": { "aws:SecureTransport": "true" } } Here's a complete example: { "Version": "2008-10-17", "Id": "some_policy", "Statement": [ { "Sid": "AddPerm", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my_bucket/*", "Condition": { "Bool": { "aws:SecureTransport": "true" } } } ] } Now accessing the contents of my_bucket over HTTP will produce a 403 error, while using HTTPS will work fine.

There are some instances when you want to store your passwords in files to be used by programs or scripts. But storing your passwords in plain text is not a good idea. Use the SecurePasswordVault to encrypt your passwords before storing and get it decrypted when you want to use it. You can use the SecurePasswordVault described here to store any number of encrypted passwords. Passwords are stored as key value pairs. Key - any name given by the user for the password Value - encrypted password SecurePasswordVault will create a file with the given name in the working directory if it doesn't exist. If a file exists then the information in that file will be read. Passwords are encrypted using the MAC address of the network card. SecurePasswordVault will use the first network card MAC which is not the loop back interface. So the encrypted file can only be decrypted with that particular MAC address. If you want to reset the pass word details, just delete the password file and run the SecurePasswordVault. You can download the sample code from the following GitHub repository https://github.com/jsdjayanga/secure_password com.wso2.devgov; import org.bouncycastle.util.encoders.Base64; import javax.crypto.*; import javax.crypto.spec.SecretKeySpec; import java.io.*; import java.net.NetworkInterface; import java.net.SocketException; import java.security.InvalidKeyException; import java.security.NoSuchAlgorithmException; import java.security.Security; import java.util.*; /** * Created by jayanga on 3/31/14. */ public class SecurePasswordVault { private static final int AES_KEY_LEN = 32; private static final int PASSWORD_LEN = 256; private static boolean initialized; private final String secureFile; private final byte[] networkHardwareHaddress; private Map secureDataMap; private List secureDataList; SecretKeySpec secretKey; public SecurePasswordVault(String filename, String[] secureData) throws IOException { Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider()); initialized = false; secureFile = filename; networkHardwareHaddress = SecurePasswordVault.readNetworkHardwareAddress(); secureDataMap = new HashMap(); this.secureDataList = new ArrayList(secureData.length); Collections.addAll(secureDataList, secureData); byte[] key = new byte[AES_KEY_LEN]; Arrays.fill(key, (byte)0); for(int index = 0; index < networkHardwareHaddress.length; index++){ key[index] = networkHardwareHaddress[index]; } secretKey = new SecretKeySpec(key, "AES"); if (!isInitialized()){ readSecureData(secureDataList); persistSecureData(); } readSecureDataFromFile(); } private boolean isInitialized(){ if (initialized == true){ return true; }else{ File file = new File(secureFile); if (file.exists()){ initialized = true; return initialized; } } return false; } private static byte[] readNetworkHardwareAddress() throws SocketException { Enumeration networkInterfaceEnumeration = NetworkInterface.getNetworkInterfaces(); if (networkInterfaceEnumeration != null){ NetworkInterface networkInterface = null; while (networkInterfaceEnumeration.hasMoreElements()){ networkInterface = networkInterfaceEnumeration.nextElement(); if (!networkInterface.isLoopback()){ break; } } if (networkInterface == null){ networkInterface = networkInterfaceEnumeration.nextElement(); } byte[] hwaddr = networkInterface.getHardwareAddress(); return hwaddr; }else{ throw new RuntimeException("Cannot initialize. Failed to generate unique id."); } } private byte[] encrypt(String word) { byte[] password = new byte[PASSWORD_LEN]; Arrays.fill(password, (byte)0); byte[] pw = new byte[0]; try { pw = word.getBytes("UTF-8"); for(int index = 0; index < pw.length; index++){ password[index] = pw[index]; } byte[] cipherText = new byte[password.length]; Cipher cipher = null; try { cipher = Cipher.getInstance("AES/ECB/NoPadding"); try { cipher.init(Cipher.ENCRYPT_MODE, secretKey); int ctLen = 0; try { ctLen = cipher.update(password, 0, password.length, cipherText, 0); ctLen += cipher.doFinal(cipherText, ctLen); return cipherText; } catch (ShortBufferException e) { e.printStackTrace(); } catch (BadPaddingException e) { e.printStackTrace(); } catch (IllegalBlockSizeException e) { e.printStackTrace(); } } catch (InvalidKeyException e) { e.printStackTrace(); } } catch (NoSuchAlgorithmException e) { e.printStackTrace(); } catch (NoSuchPaddingException e) { e.printStackTrace(); } } catch (UnsupportedEncodingException e) { e.printStackTrace(); } return null; } private String decrypt(byte[] cipherText) { byte[] plainText = new byte[PASSWORD_LEN]; Cipher cipher = null; try { cipher = Cipher.getInstance("AES/ECB/NoPadding"); try { cipher.init(Cipher.DECRYPT_MODE, secretKey); int plainTextLen = 0; try { plainTextLen = cipher.update(cipherText, 0, PASSWORD_LEN, plainText, 0); try { plainTextLen += cipher.doFinal(plainText, plainTextLen); String password = new String(plainText); return password.trim(); } catch (IllegalBlockSizeException e) { e.printStackTrace(); } catch (BadPaddingException e) { e.printStackTrace(); } } catch (ShortBufferException e) { e.printStackTrace(); } } catch (InvalidKeyException e) { e.printStackTrace(); } } catch (NoSuchAlgorithmException e) { e.printStackTrace(); } catch (NoSuchPaddingException e) { e.printStackTrace(); } return null; } public void readSecureData(List secureDataList) throws IOException { BufferedReader bufferRead = new BufferedReader(new InputStreamReader(System.in)); for(int index = 0; index < secureDataList.size(); index++){ System.out.println("Please enter the value for :" + secureDataList.get(index)); String value = new String(Base64.encode(encrypt(bufferRead.readLine()))); secureDataMap.put(secureDataList.get(index), value); } } public String getSecureData(String key) { String value = secureDataMap.get(key); if (value != null){ return decrypt(Base64.decode(value.getBytes())); } throw new RuntimeException("Given key is unknown. [key=" + key + "]"); } private void readSecureDataFromFile() throws IOException { BufferedReader br = new BufferedReader(new FileReader(secureFile)); String line; while ((line = br.readLine()) != null){ int dividerPoint = line.indexOf("="); if (dividerPoint > 0){ secureDataMap.put(line.substring(0, dividerPoint), line.substring(dividerPoint + 1)); } } } private void persistSecureData() throws IOException { FileWriter fileWriter = new FileWriter(secureFile); for(String key : secureDataMap.keySet()){ fileWriter.append(key + "=" + secureDataMap.get(key) + "\n"); } fileWriter.close(); } }

If you use Java for programming, read on to learn about tools like Eclipse IDE, the Java Development Kit, and other must-know tools.

A recurring customer issue seen by the Percona Support team involves how to make the ibdata1 file “shrink” within MySQL. I'll show you how to handle big tables.

Originally written by Virgil Dodson Actuate recently released a new, free BIRT server called the BIRT iHub F-Type. It incorporates all the functionality of BIRT iHub and is limited only by the capacity of output it can deliver on a daily basis. It is ideal for departmental and smaller scale applications. When BIRT F-Type reaches its maximum output capacity, additional capacity can be purchased on a subscription based model. Some of the key features of BIRT iHub F-Type that will help improve your BIRT content applications are: Interactivity – Allow end-users to modify and personalize reports, and answer questions themselves. Scheduling – Automate report generation based on rules and calendar, and then notify users. Sharing – Secure document management and distribution that allows users to only access content/data they are entitled to. Excel Emitter – Export as native Excel (not CSV) with formulas/pivot tables/worksheets/charts. Integration – JavaScript API to embed dynamic reports and visualizations in your web app. Downloading BIRT iHub F-Type Before we get started with the installation process, we need to download BIRT iHub F-Type. There are three downloads available: Windows, Linux, and a VMware image. This blog will cover the Windows installation. If you’re installing either of the other types, you’ll find links to guides for them at the bottom of this blog post. Once you click on your chosen download, you’ll be asked to register. If you’ve already registered, click the “Click to Login” button. If not, fill out the short registration form to get started. Next, read and accept the license agreement. Once you’ve done that, click the checkbox, and a link for the download will appear. Click that to start your download. At this point, you should also receive an email with an activation code. Be sure to check your spam folder if you don’t see it in your inbox. Installing BIRT iHub F-Type After the download is complete, launch the executable file named ActuateBIRTiHubFType.exe. A welcome message will appear. Press Next to continue. You must read and accept the license agreement on the next screen. Choose a destination folder for the installation. The default is C:\Actuate\BIRTiHub. If you have existing BIRT designs that depend on a JDBC database driver, you can optionally specify the folder where these drivers are located. Press Next to continue. Once the installation has finished, press Finish to launch the BIRT iHub F-Type. A desktop shortcut is also created that points to the iHub F-Type URL at http://localhost:8700/iportal. The first time you launch the BIRT iHub F-Type, you will need to activate it. Enter the activation code that you should have received in an e-mail. After entering a valid activation code, you should receive a message that the code was accepted and the BIRT iHub F-Type should start initializing services. Once that has completed, you will be presented with the login screen. The default user name is “administrator” and the password is blank for your first log in. You’ll be able to change this after you have logged in. Press “Log In” to continue. The first time you launch the BIRT iHub F-Type, you will be in tutorial mode which will help you get started loading your BIRT content and required resources. You can bypass the tutorial mode at any time by pressing the “Exit Tutorial” button at the top right. Select a BIRT design (*.rptdesign) file and press the Upload button. If you don’t have a BIRT design, you can download a sample from the link on the same page. The BIRT design file is automatically inspected and if there are any dependent files needed, like images, data files, BIRT report libraries, CSS styles, or other linked BIRT designs, you will be asked to upload those files as well. Once your BIRT design and dependent files are uploaded, your BIRT report will be displayed in the BIRT iHub F-Type and is now ready to explore. Thanks for reading. Now, it’s time to unleash the full power of BIRT into your application. If you have any questions or comments, please feel free to use the comments section below or visit the BIRT iHub F-Type forum. -Virgil For more blogs in the “Introducing BIRT iHub F-Type” series, see the list below: Installing iHub F-Type: Linux | VMWare Image

The NetBeans Team has released NetBeans IDE 8.0.1, with significant enhancements to features relating to HTML5, JavaScript, and CSS3. An update to NetBeans IDE 8.0, this release includes the following notable changes: Modularity and enterprise JavaScript development support via integration of RequireJS Tools for working seamlessly with Grunt, Karma, and Bower Enhanced Java Editor, PHP Editor, and Git Tools Support for new versions of WebLogic, GlassFish, Tomcat, WildFly, and PrimeFaces Performance improvements and bug fixes Complete list of features: http://wiki.netbeans.org/NewAndNoteworthyNB801 To get the recent changes: Download and install NetBeans 8.0.1, which includes the recently released GlassFish 4.1 OR If you already have NetBeans IDE 8.0 installed, launch the IDE and an update notification will appear. Alternatively, choose Help | Check for Updates. Click the alert-box to install the updates. For details on upgrading, see the YouTube screencast "How to Upgrade to NetBeans IDE 8.0.1 from NetBeans IDE 8.0" on the NetBeans YouTube channel. NetBeans IDE 8.0.1 is available in English, Brazilian Portuguese, Japanese, Russian, and Simplified Chinese. We welcome feedback about your use of NetBeans software. Share your thoughts on the NetBeans mailing lists and forums; and keep track of NetBeans news by subscribing to the NetBeans Weekly Newsletter and following NetBeans on Twitter, Facebook, and YouTube.