At Bloomfire’s User Conference in May, I had the pleasure of listening to City of Austin’s Chief Innovation Officer Kerry O’Connor present on how government knowledge management is changing. The Innovation Office focuses on internal and public service innovation, as well as open government. O’Connor has worked in the public sector for many years – at the U.S. Department of State, the Office of Management Policy Rightsizing and Innovation, and several U.S. Embassies. She talked about seeing firsthand that the government is changing from a “need to know organization” to a “need to share organization.” O’Connor argues that disruption is inevitable, and will come whether in the form of opportunity or threat – and there’s no script. “When there’s no script,” O’Connor says, “we have to be intentionally improvisational.” O’Connor defines innovation as any project that is new to you and has an uncertain outcome. She talked about how important knowledge is in supporting innovation. As the first person to ever fill this role, her goal for her first year in office was to set up an innovation infrastructure. This included putting into place the processes, teams, and skills and information to create an environment that fosters innovation. O’Connor recommends that to facilitate intentional improvisation, you must frame the problems you want to solve first. Once you know the goal, look for innovation technology infrastructure that helps you manage contacts, relationships, projects, knowledge, ideas, and insights. We live in a world that is increasingly interconnected and disrupted, and O’Connor says that organizations are naturally becoming more networked, human-centered, and improvisational. She encouraged attendees to “use what you have; we must connect, coach, mentor, share, and experiment.” To ensure that citizens can interact with the knowledge that city employees have, the City of Austin created online public spaces. These spaces, created on Bloomfire, offer the opportunity for citizens to participate in a conversation with employees around innovation, data, and city orientation. I was inspired by O’Connor’s presentation, and proud to live in a city that is so forward thinking about how information is shared. It made me want to get more involved in finding ways to solve some of problems Austin is facing as a result of our rapid growth. As a result of her talk, I’m going to try to make it to this weekend’s ATX Hack for Change. If you would like to watch O’Connor’s entire presentation, you can access it on the Bloomfire Community. Like this post? Click here to subscribe to our blog and receive the latest content on social learning, customer support, sales enablement, or all three.

I was just asked for 5 tips on reducing memory footprint in background tasks. Here's what I came up with: 1. Include as few external dependencies as possible. 2. Include as little of your own code as possible. 3. Don't load anything in memory you don't need - and free anything you do load before carrying on. 4. Do as little as possible - stick tightly to only what you absolutely must do. 5. Don't be afraid to split functionality between multiple tasks if need be. Yes, they're all quite generic but this is actually a fairly standard scenario on mobile. Ultimately, to save time and resources (memory, network, power, etc.) you should do as little as possible and only use the things you must.

Written by Jim Magers for LeadingAgile. When I’m teaching an agile bootcamp class and talking about work in process, I always make a point (usually multiple times) to tell the attendees that agile is about finishing work…not about starting work. I reinforce this by pointing out that you can have a glorious looking burndown chart for the duration of the sprint but completely fail in your mission to meet your commitments and finish stories. The team can be burning down hours beautifully on a daily basis, with the remaining task hours looking like they are tracking right along the ideal line, and then boom… It’s closing time for the sprint and no stories actually got completed. Remember that notion of building working, tested software? Didn’t happen. The team started too many stories at once and ended up not being able to bring any of them across the finish line when the bell rang. This notion of finishing work applies to sprint planning as well. If you short-change the time it takes to do good sprint planning, and the team meanders off to begin writing code and test plans too soon, there is a risk that the team is going to struggle to be successful. Remember what we do in sprint planning. Consider velocity, and load the sprint backlog with high priority stories from the product backlog. Check. Determine capacity for the team to work on sprint tasks over the coming sprint. Check. Break stories into tasks and determine who is doing what. Check. Make sure the work is going to fit. Check. Commit to the work. Check. But what can happen when you don’t take the time to thoughtfully break down tasks and estimate hours of effort? Consider the following burn down chart. In this example, the team left sprint planning thinking that they were committing to 830 hours. But just two days into the sprint they discovered additional task hours and instead found themselves in the awkward position of actually needing almost 1200 hours of capacity to complete the committed stories. Guess what…they did not have 1200 hours of capacity to give, especially since they were now a full two days into the work. In looking more closely at their burndown chart over the course of the two-week sprint, it took them almost 6 days of work to get back to the point where they had 830 hours of work left to do. Six days just to get back to where they thought was their starting point when they concluded their sprint planning meeting. And surprise, they didn’t finish the sprint successfully. So, don’t short-change the value that good sprint planning affords. Yes, it takes time. Yes, it can seem tedious. Yes, the team is anxious to get started. But good sprint planning pays dividends. Remember that it is about finishing work, and not about starting work.

As a continuous delivery service, we form an integral part of our customers’ workflow. Any interruption to our service impacts our customers a lot, so we need to make sure we’re resolving any service interruption immediately. By building our system to be very resilient with immutable infrastructure at the core, we’re able to react to most interruptions quickly. Additionally we’ve implemented automatic health checks to remove systems from production as soon as the load or network latency becomes too high. This is very important when deploying into cloud systems because the cloud behaves more unpredictably than your own hardware. You want to take this into account when building your system so your team can be focused on building your product at all times. A recent interview I did with Statuspage prompted me to write in more detail about how we respond to incidents. In this post, I’ll outline our processes, tools, and the document templates we use to make sure we have a repeatable workflow. Downtime Recognition Downtime recognition must always be based on metrics in your system. If downtimes are only recognized when your team or customers see it in the production application a negative impact has already happened. To make sure we’ve got great insights into our infrastructure, we use Librato Metrics, Pingdom, and New Relic for collecting various application and server metrics. Every system has various alerts set that trigger Pagerduty incidents. Our on-call hours We use Pagerduty to manage on-call times and notify our team of any service interruption. We’re spread out between Vienna and Boston and a few other remote locations, so we can cover almost 24 hours without getting into somebody’s night time. A person woken up in the middle of the night will not perform at their best during the outage. Plus, it ruins their productivity for the next day. Our local coverage times are as follows: Europe: 4 a.m. – 4 p.m. CEST Boston: 10 a.m. – 10 p.m. EST Both are reasonable times for each city. In the worst case, somebody might be woken up in the morning in Vienna, though that’s at a time where we get much less traffic, so service interruptions are less likely. We can cover the whole US workday, from East to West Coast, from Boston so nobody has to be woken up during this most crucial time of the day for us. We’ve recently switched to weekly schedules, so you’re on call for a week in your timezone. Weekends are covered completely by one person, so it’s rare that you have to be on call on a weekend. To make sure there’s always somebody covering even if somebody misses a pager call, we’ve set up a secondary schedule where even non-technical people are on call. Our primary versus secondary schedules The primary schedule is always comprised of people who can fix our infrastructure; the secondary is made up of most of the rest of the team. This has several advantages. First of all, we share on-call duty between most people in the team. Even if one of the developers doesn’t see the page, somebody else picks up and can ping other developers and talk to customers. Having only the development team on a schedule would put an unfair burden on the dev team. It would also make it more difficult to resolve the issue and communicate with customers at the same time. Through this secondary schedule system, we always have somebody we can pull in to take over customer communication. That can mean calling the person on secondary support at night as well if that’s necessary. It usually isn’t necessary to pull in the secondary, but it’s definitely a great fallback to have. In case a pager call falls through the first and second layer, Jim, our VPE, is on a third layer, and I am on a fourth layer. This makes sure the escalation goes all through the technical team in Codeship if necessary (thankfully it hasn’t been for a long time). While growing our team, we’re constantly iterating on our coverage hours to make sure we have great coverage at all times. Downtime Remediation and Follow Up As soon as the issue is discovered, the development team will decide who will look into it. Our customer success team will decide who will take over communication for this specific issue and will work with the assigned developer to communicate regularly with our customers. By defining those two people, we have a clear line of communication between the developers who know the current status of the issue and customer success who can answer support requests and send updates through Twitter or our Statuspage. Steps while fixing the issue The first and most important step is to update our Statuspage. We want to make sure we’re always communicating immediately with our customers if we have a service interruption. We’re always trying to ask ourselves, “How would we expect important services that we use to communicate with us in a downtime?” and “How can we build more long-term trust through our downtime communication?” The way you communicate to your customers during a downtime will decide if they trust you more or less in the future. A single interruption doesn’t necessarily erode trust, but a downtime with bad communication definitely will. Customers expect full transparency when issues are coming up, including a detailed description of what the issue is and how it will be resolved. While fixing the issue, we will regularly update our customers through various channels, and the development team and customer success are in constant contact with each other. As we’re currently growing our engineering team, we will probably move to a setup where the roles are even more clearly defined. As soon as you have several teams working on different services where downtime might impact several services at once, you need to have a higher level management role in place to manage the downtime effort. Heroku released a very interesting post last year about their incident response with the Incident Command System. It’s definitely an interesting approach that we’re thinking about for the future. Debriefings As soon as an issue is fixed, we start work on a debriefing about the service interruption. We’ve found structured debriefings to be an incredibly helpful tool to get an overview of the whole issue and determine future steps we need to take. Because we’re spread out between Europe and the US, compiling this debrief is typically done asynchronously on GitHub with people commenting or committing and pushing changes directly onto the debriefing branch. Meetings or calls are done if necessary, but typically don’t have to be part of a debriefing. We have a GitHub repository called Operations that contains our debriefings as well as a template for them. I’ve copied the template into a GitHub Gist. The template contains an executive summary so anyone can get a quick overview on what happened. We can share this summary internally. The debriefing template then dives into different aspects of the issue, from detailed technical descriptions to customer communication. We also capture any steps we can take to improve our resolution efforts in the future. We want to make sure this document accurately reflects what has happened and digs deep while at the same time doesn’t push blame on anyone. Only when we analyze exactly what happened in our technology and process are we able to improve. But we won’t be able to dig deep and make sure everyone opens up about their mistakes if we then turn around and immediately use it to punish people. One very important lesson we’ve learned about our template is to leave irrelevant sections of the template empty instead of removing them for a particular debriefing. Removing irrelevant sections makes it difficult to compare debriefings to one another. It can also cause somebody not directly involved with the downtime to wonder if something is missing or getting hidden. Debriefings should be as transparent as possible to maintain your team’s trust in the process. The debriefings also capture To Dos, so it’s easy for anyone to follow up and see if we followed through with steps we needed to improve. These debriefings also form the core data we use to write any public follow-up blog post or any other customer communication we’re doing after the resolution. Conclusions By having a well-defined process that goes from detection of an issue to customer communication to solving the issue to debriefing, we can focus all of our energy on resolving an issue once it comes up instead of scrambling to find an ad hoc process. In fact, we want this process to be so second nature to everyone that we wrote down a document outlining those steps and hung them up in some of our restrooms. You can download the document here. We also include the document as part of our onboarding activities. This ensures that everyone who joins knows exactly how to resolve an issue once they become part of the Pagerduty schedule. While we’re definitely going to change and formalize our process as we grow, we feel we’ve got the right balance of process and flexibility for the current size of our team. Let us know if you have a different process for finding or resolving issues in your system in the comments.

this afternoon i got a review copy of the book creating symmetry: the artful mathematics of wallpaper patterns . here’s a striking curves from near the beginning of the book, one that the author calls the “mystery curve.” the curve is the plot of exp( it ) – exp(6 it )/2 + i exp(-14 it )/3 with t running from 0 to 2π. here’s python code to draw the curve. import matplotlib.pyplot as plt from numpy import pi, exp, real, imag, linspace def f(t): return exp(1j*t) - exp(6j*t)/2 + 1j*exp(-14j*t)/3 t = linspace(0, 2*pi, 1000) plt.plot(real(f(t)), imag(f(t))) # these two lines make the aspect ratio square fig = plt.gcf() fig.gca().set_aspect('equal') plt.show() maybe there’s a more direct way to plot curves in the complex plane rather than taking real and imaginary parts. updated code for the aspect ratio per janne’s suggestion in the comments. related posts : several people have been making fun visualizations that generalize the example above. brent yorgey has written two posts, one choosing frequencies randomly and another that animates the path of a particle along the curve and shows how the frequency components each contribute to the motion. mike croucher developed a jupyter notebook that lets you vary the frequency components with sliders. john golden created visualizations in geogerba here and here . jennifer silverman showed how these curves are related to decorative patterns that popular in the 1960’s. she also created a coloring book and a video . dan anderson accused me of nerd sniping him and created this visualization .

From time to time I’ve been heard to say: “Agile is Punk.” But I’ve never explained myself. I’ve also been heard to say things life “Agile is about democratising the workplace” but I’ve never explain myself there either. Let me try… What I mean when I say this is: Agile (software development) has a lot in common with Punk rock. To me the important thing about punk rock was that it was about people trying it - music, their own thing. Anybody trying to play music, anybody forming a band, anyone who had a novel idea trying to get a record contract. Yes, even if they couldn’t play an instrument they could have a go, and who knows, maybe they would learn as they went. (I should say that while I’m old enough to have been around when punk was I’m not old enough to have been there. Both post-punk and post-disco influences were at work in the New Wave music which was common when I came music listening.) Punk had a democratising effect on music. Music has aways been of the people, anyone can listen, anyone can try to sing - although I’m not very good at it even I can try! But the music industry was something different, performing, recording - there were barriers there! Punk tore down barriers. Punk opened up the recording industry. Punk opened up music. Agile opened up the software process industry. Before Agile official software processes were pretty locked down. You had to be an academic or expert/consultant to dabble in that space. Programmers who worked in under official software processes were on the receiving end. Agile said: “Your opinion is important too.” In truth music has always was been open to everyone, just not the recording industry. And in software development processes were open to anyone, most programmers did not work under an official process, mostly it was common practices which, if they worked, were probably more effective than official ones. Unfortunately these unofficial work practices came with guilt: because we did not do it the way the books said we should. When faults occurred we blame ourselves for “not doing it properly”. Agile says: “Everyone involved with software development should have a voice in deciding how to work, it can be improved and you don’t need to be an expert, academic, consultant or certified member of some body to express that view.” That also makes it democratic. I don’t mean democratic in the sense that we all get to vote, I mean democratic in the sense that it is power is vested in the hands of the collective people. Everyone has a voice, everyone can participate, and those who hold executive power do so by the will of the people. Agile is about giving everyone a voice. Like Punk that means accept that those who don’t have much skill are also entitled to a voice. Funnily enough, I’ve long held that any punk band that made a second album weren’t punk anymore because they were part of the industry, they were now experts! The same is true with Agile, hang around for long enough (like me) and you are no longer an outside but an insider, an expert. Increasingly we see Agile heading outside of software development. When this happens it becomes necessary to ask: What is Agile? My answer is: Democracy. Agile is about valuing everyone, agile is about giving everyone a voice, agile is about putting the power to change the workplace (process, systems, norms) into the collective hands of the people who work there. Yes at times it feels revolutionary, but there are fellow travellers, it is all a Theory-Y movement.

Haskell advocates are fond of saying that a Haskell function cannot launch missiles without you knowing it. Pure functions have no side effects, so they can only do what they purport to do. In a language that does not enforce functional purity, calling a function could have arbitrary side effects, including launching missiles. But this cannot happen in Haskell. The difference between pure functional languages and traditional imperative languages is not quite that simple in practice. Programming with pure functions is conceptually easy but can be awkward in practice. You could just pass each function the state of the world before the call, and it returns the state of the world after the call. It’s unrealistic to pass a program’s entire state as an argument each time, so you’d like to pass just that state that you need to, and have a convenient way of doing so. You’d also like the compiler to verify that you’re only passing around a limited slice of the world. That’s where monads come in. Suppose you want a function to compute square roots and log its calls. Your square root function would have to take two arguments: the number to find the root of, and the state of the log before the function call. It would also return two arguments: the square root, and the updated log. This is a pain, and it makes function composition difficult. Monads provide a sort of side-band for passing state around, things like our function call log. You’re still passing around the log, but you can do it implicitly using monads. This makes it easier to call and compose two functions that do logging. It also lets the compiler check that you’re passing around a log but not arbitrary state. A function that updates a log, for example, can effect the state of the log, but it can’t do anything else. It can’t launch missiles. Once monads get large and complicated, it’s hard to know what side effects they hide. Maybe they can launch missiles after all. You can only be sure by studying the source code. Now how do you know that calling a C function, for example, doesn’t launch missiles? You study the source code. In that sense Haskell and C aren’t entirely different. The Haskell compiler does give you assurances that a C compiler does not. But ultimately you have to study source code to know what a function does and does not do.

Written by Norberto Leite Introduction In this 2-part blog series, you will learn how to use MongoDB, Mongoose Object Data Mapping (ODM) with Express.js and Node.js. These technologies use a uniform language - JavaScript - providing performance gains in the software and productivity gains for developers. In this first part, we will describe the basic mechanics of our application and undertake data modeling. In the second part, we will create tests that validate the behavior of our application and then describe how to set-up and run the application. No prior experience with these technologies is assumed and developers of all skill levels should benefit from this blog series. So, if you have no previous experience using MongoDB, JavaScript or building a REST API, don’t worry - we will cover these topics with enough detail to get you past the simplistic examples one tends to find online, including authentication, structuring code in multiple files, and writing test cases. Let’s begin by defining the MEAN stack. What is the MEAN stack? The MEAN stack can be summarized as follows: M = MongoDB/Mongoose.js: the popular database, and an elegant ODM for node.js. E = Express.js: a lightweight web application framework. A = Angular.js: a robust framework for creating HTML5 and JavaScript-rich web applications. N = Node.js: a server-side JavaScript interpreter. The MEAN stack is a modern replacement for the LAMP (Linux, Apache, MySQL, PHP/Python) stack that became the popular way for building web applications in the late 1990s. In our application, we won’t be using Angular.js, as we are not building an HTML user interface. Instead, we are building a REST API which has no user interface, but could instead serve as the basis for any kind of interface, such as a website, an Android application, or an iOS application. You might say we are building our REST API on the ME(a)N stack, but we have no idea how to pronounce that! What is a REST API? REST stands for Representational State Transfer. It is a lighter weight alternative to SOAP and WSDL XML-based API protocols. REST uses a client-server model, where the server is an HTTP server and the client sends HTTP verbs (GET, POST, PUT, DELETE), along with a URL and variable parameters that are URL-encoded. The URL describes the object to act upon and the server replies with a result code and valid JavaScript Object Notation (JSON). Because the server replies with JSON, it makes the MEAN stack particularly well suited for our application, as all the components are in JavaScript and MongoDB interacts well with JSON. We will see some JSON examples later, when we start defining our Data Models. The CRUD acronym is often used to describe database operations. CRUD stands for CREATE, READ, UPDATE, and DELETE. These database operations map very nicely to the HTTP verbs, as follows: POST: A client wants to insert or create an object. GET: A client wants to read an object. PUT: A client wants to update an object. DELETE: A client wants to delete an object. These operations will become clear later when define our API. Some of the common HTTP result codes that are often used inside REST APIs are as follows: 200 - “OK”. 201 - “Created” (Used with POST). 400 - “Bad Request” (Perhaps missing required parameters). 401 - “Unauthorized” (Missing authentication parameters). 403 - “Forbidden” (You were authenticated but lacking required privileges). 404 - “Not Found”. A complete description can be found in the RFC document, listed in the resources section at the end of this blog. We will use these result codes in our application and you will see some examples shortly. Why Are We Starting with a REST API? Developing a REST API enables us to create a foundation upon which we can build all other applications. As previously mentioned, these applications may be web-based or designed for specific platforms, such as Android or iOS. Today, there are also many companies that are building applications that do not use an HTTP or web interface, such as Uber, WhatsApp, Postmates, and Wash.io. A REST API also makes it easy to implement other interfaces or applications over time, turning the initial project from a single application into a powerful platform. Creating our REST API The application that we will be building will be an RSS Aggregator, similar to Google Reader. Our application will have two main components: The REST API Feed Grabber (similar to Google Reader) In this blog series we will focus on building the REST API, and we will not cover the intricacies of RSS feeds. However, code for Feed Grabber is available in a github repository, listed in the resources section of this blog. Let’s now describe the process we will follow in building our API. We will begin by defining the data model for the following requirements: Store user information in user accounts Track RSS feeds that need to be monitored Pull feed entries into the database Track user feed subscriptions Track which feed entry a user has already read Users will need to be able to do the following: Create an account Subscribe/unsubscribe to feeds Read feed entries Mark feeds/entries as read or unread Modeling Our Data An in-depth discussion on data modeling in MongoDB is beyond the scope of this article, so see the references section for good resources on this topic. We will need 4 collections to manage this information: Feed collection Feed entry collection User collection User-feed-entry mapping collection Let’s take a closer look at each of these collections. Feed Collection Lets now look at some code. To model a feed collection, we can use the following JSON document: If you are familiar with relational database technology, then you will know about databases, tables, rows and columns. In MongoDB, there is a mapping to most of these Relational concepts. At the highest level, a MongoDB deployment supports one or more databases. A database contains one or more collections, which are the similar to tables in a relational database. Collections hold documents. Each document in a collection is, at a highest level, similar to a row in a relational table. However, documents do not follow a fixed schema with pre-defined columns of simple values. Instead, each document consists of one or more key-value pairs where the value can be simple (e.g., a date), or more sophisticated (e.g., an array of address objects). Our JSON document above is an example of one RSS feed for the Eater Blog, which tracks information about restaurants in New York City. We can see that there are a number of different fields but the key ones that our client application may be interested in include the URL of the feed and the feed description. The description is important so that if we create a mobile application, it would show a nice summary of the feed. The remaining fields in our JSON document are for internal use. A very important field is _id. In MongoDB, every document must have a field called _id. If you create a document without this field, at the point where you save the document, MongoDB will create it for you. In MongoDB, this field is a primary key and MongoDB will guarantee that within a collection, this value is unique. Feed Entry Collection After feeds, we want to track feed entries. Here is an example of a document in the feed entry collection: Again, we can see that there is a _id field. There are also some other fields, such as description, title and summary. For the content field, note that we are using an array, and the array is also storing a document. MongoDB allows us to store sub-documents in this way and this can be very useful in some situations, where we want to hold all information together. The entryID field uses the tag format to avoid duplicate feed entries. Notice also the feedID field that is of type ObjectId - the value is the _id of the Eater Blog document, described earlier. This provides a referential model, similar to a foreign key in a relational database. So, if we were interested to see the feed document associated with this ObjectId, we could take the value 523b1153a2aa6a3233a913f8 and query the feed collection on _id, and it would return the Eater Blog document. User Collection Here is the document we could use to keep track of users: A user has an email address, first name and last name. There is also an sp_api_key_id and sp_api_key_secret - we will use these later with Stormpath, a user management API. The last field, called subs, is a subscription array. The subs field tells us which feeds this user is subscribed-to. User-Feed-Entry Mapping Collection The last collection allows us to map users to feeds and to track which feeds have been read. We use a Boolean (true/false) to mark the feed as read or unread. Functional Requirements for the REST API As previously mentioned, users need to be able to do the following: Create an account. Subscribe/unsubscribe to feeds. Read feed entries. Mark feeds/entries as read or unread. Additionally, a user should be able to reset their password. The following table shows how these operations can be mapped to HTTP routes and verbs. Route Verb Description Variables /user/enroll POST Register a new user firstName lastName email password /user/resetPassword PUT Password Reset email /feeds GET Get feed subscriptions for each user with description and unread count /feeds/subscribe PUT Subscribe to a new feed feedURL /feeds/entries GET Get all entries for feeds the user is subscribed to /feeds/&ltfeedid>/entries GET Get all entries for a specific feed /feeds/&ltfeedid> PUT Mark all entries for a specific feed as read or unread read = &lttrue | false> /feeds/&ltfeedid>/entries/&ltentryid> PUT Mark a specific entry as either read or unread read = &lttrue | false> /feeds/&ltfeedid> DELETE Unsubscribe from this particular feed In a production environment, the use of secure HTTP (HTTPS) would be the standard approach when sending sensitive details, such as passwords. Real World Authentication with Stormpath In robust real-world applications it is important to provide user authentication. We need a secure approach to manage users, passwords, and password resets. There are a number of ways we could authenticate users for our application. One possibility is to use Node.js with the Passport Plugin, which could be useful if we wanted to authenticate with social media accounts, such as Facebook or Twitter. However, another possibility is to use Stormpath. Stormpath provides User Management as a Service and supports authentication and authorization through API keys. Basically, Stormpath maintains a database of user details and passwords and a client application REST API would call the Stormpath REST API to perform user authentication. The following diagram shows the flow of requests and responses using Stormpath. In detail, Stormpath will provide a secret key for each “Application” that is defined with their service. For example, we could define an application as “Reader Production” or “Reader Test”. This could be very useful when we are still developing and testing our application, as we may be frequently adding and deleting test users. Stormpath will also provide an API Key Properties file. Stormpath also allows us to define password strength requirements for each application, such as: Must have >= 8 characters. Must include lowercase and uppercase. Must include a number. Must include a non-alphabetic character Stormpath keeps track of all of our users and assigns them API keys, which we can use for our REST API authentication. This greatly simplifies the task of building our application, as we don’t have to focus on writing code for authenticating users. Node.js Node.js is a runtime environment for server-side and network applications. Node.js uses JavaScript and it is available for many different platforms, such as Linux, Microsoft Windows and Apple OS X. Node.js applications are built using many library modules and there is a very rich ecosystem of libraries available, some of which we will use to build our application. To start using Node.js, we need to define a package.json file describing our application and all of its library dependencies. The Node.js Package Manager installs copies of the libraries in a subdirectory, called node_modules/, in the application directory. This has benefits, as it isolates the library versions for each application and so avoids code compatibility problems if the libraries were to be installed in a standard system location, such as /usr/lib, for example. The command npm install will create the node_modules/ directory, with all of the required libraries. Here is the JavaScript from our package.json file: Our application is called reader-api. The main file is called server.js. Then we have a list of the dependent libraries and their versions. Some of these libraries are designed for parsing the HTTP queries. The test harness we will use is called frisby. The jasmine-node is used to run frisby scripts. One library that is particularly important is async. If you have never used node.js, it is important to understand that node.js is designed to be asynchronous. So, any function which does blocking input/output (I/O), such as reading from a socket or querying a database, will take a callback function as the last parameter, and then continue with the control flow, only returning to that callback function once the blocking operation has completed. Let’s look at the following simple example to demonstrate this. In the above example, we may think that the output would be: one two but in fact it might be: two one because the line that prints “one” might happen later, asynchronously, in the callback. We say “might” because if conditions are just right, “one” might print before “two”. This element of uncertainty in asynchronous programming is called non-deterministic execution. For many programming tasks, this is actually desirable and permits high performance, but clearly there are times when we want to execute functions in a particular order. The following example shows how we could use the async library to achieve the desired result of printing the numbers in the correct order: In the above code, we are guaranteed that function two will only be called after function one has completed. Wrapping Up Part 1 Now that we have seen the basic mechanics of node.js and async function setup, we are ready to move on. Rather than move into creating the application, we will instead start by creating tests that validate the behavior of the application. This approach is called test-driven development and has two very good features: It helps the developer really understand how data and functions are consumed and often exposes subtle needs like the ability to return 2 or more things in an array instead of just one thing. By writing tests before building the application, the paradigm becomes “broken / unimplemented until proven tested OK” instead of “assumed to be working until a test fails.” The former is a “safer” way to keep the code healthy.

Introduction As we all know about the Identity columns and Merge statement. We are not going to discuss any boring theoretical tropics related to it. Better we are discussing here with a practical scenario of merging records. Hope all of you must enjoy it and it will be informative. The Scenario We have a Table with Identity Columns named #tbl_TempStudentRecords. The table details are mentioned bellow. Column Name SrlNo StudentName StudentClass StudentSection We have another table named #tbl_TempStudrntMsrks. The table details are mentioned bellow. Column Name SrlNo StubjectName MarksObtain What we want to do is, we have another set of table called tbl_StudentDetails mentioned bellow. Column Name StdRoll (PK) StudentName StudentClass StudentSection Another table named tbl_StudentMarks Column Name IdNo (PK) StdRoll (FK) References [tbl_StudentDetails].[StdRoll] SubjectName MarksObtain Her we can insert records very easily in tbl_StudentDetails from #tbl_TempStudentRecords very easily. But the main problem is the IDENTITY columns in the Table named [tbl_StudentDetails].[ StdRoll]. When we insert records the Identity columns values generate automatically. When we are trying to insert records into the table named tbl_StudentMarks from Table named #tbl_TempStudrntMsrks we have to provide the StdRoll values, which is the Foreign Key References to the Table named [tbl_StudentDetails].[ StdRoll]. Think one minute with the case scenario. Hope you can understand the problem. Now we have to solve it and we are not using any LOOP for that and NOT even any DDL operation to change the structure of base table. We are just using the SET BASED operation to make performance high. How to Solve it Step – 1 [ Create the Base Table First ] IF OBJECT_ID('tempdb..#tbl_TempStudentRecords')IS NOT NULL BEGIN DROP TABLE #tbl_TempStudentRecords; END GO CREATE TABLE #tbl_TempStudentRecords ( SrlNo BIGINT NOT NULL, StudentName VARCHAR(50) NOT NULL, StudentClass INT NOT NULL, StudentSection CHAR(1) NOT NULL ); GO IF OBJECT_ID('tempdb..#tbl_TempStudrntMsrks')IS NOT NULL BEGIN DROP TABLE #tbl_TempStudrntMsrks; END GO CREATE TABLE #tbl_TempStudrntMsrks ( SrlNo BIGINT NOT NULL, StubjectName VARCHAR(50) NOT NULL, MarksObtain INT NOT NULL ); GO IF OBJECT_ID(N'[dbo].[tbl_StudentDetails]', N'U')IS NOT NULL BEGIN DROP TABLE [dbo].[tbl_StudentDetails]; END GO CREATE TABLE [dbo].[tbl_StudentDetails] ( StdRoll BIGINT NOT NULL IDENTITY(100,1) PRIMARY KEY, StudentName VARCHAR(50) NOT NULL, StudentClass INT NOT NULL, StudentSection CHAR(1) NOT NULL ); GO IF OBJECT_ID(N'[dbo].[tbl_StudentMarks]', N'U')IS NOT NULL BEGIN DROP TABLE [dbo].[tbl_StudentMarks]; END GO CREATE TABLE [dbo].[tbl_StudentMarks] ( IdNo BIGINT NOT NULL IDENTITY(1,1) PRIMARY KEY, StdRoll BIGINT NOT NULL, SubjectName VARCHAR(50) NOT NULL, MarksObtain INT NOT NULL ); GO ALTER TABLE [dbo].[tbl_StudentMarks] ADD CONSTRAINT FK_StdRoll_tbl_StudentMarks FOREIGN KEY(StdRoll) REFERENCES [dbo].[tbl_StudentDetails](StdRoll); Step – 2 [ Inserting Records in Temp Table ] INSERT INTO #tbl_TempStudentRecords (SrlNo, StudentName, StudentClass, StudentSection) VALUES(1, 'Joydeep Das', 1, 'A'), (2, 'Preeti Sharma', 1, 'A'), (3, 'Deepasree Das', 1, 'A'); INSERT INTO #tbl_TempStudrntMsrks (SrlNo, StubjectName, MarksObtain) VALUES (1, 'Bengali', 50), (1, 'English', 70), (1, 'Math', 80), (2, 'Bengali', 0), (2, 'English', 70), (2, 'Math', 80), (3, 'Bengali', 20), (3, 'English', 90), (3, 'Math', 95); Step – 3 [ Now Solve it By MERGE Statement ] BEGIN DECLARE @MappingTable TABLE ([NewRecordID] BIGINT, [OldRecordID] BIGINT) MERGE [dbo].[tbl_StudentDetails] AS target USING (SELECT [SrlNo] AS RecordID_Original ,[StudentName] ,[StudentClass] ,[StudentSection] FROM #tbl_TempStudentRecords ) AS source ON (target.StdRoll = NULL) WHEN NOT MATCHED THEN INSERT ([StudentName], [StudentClass], [StudentSection]) VALUES (source.[StudentName],source.[StudentClass], source.[StudentSection]) OUTPUT inserted.[StdRoll], source.[RecordID_Original] INTO @MappingTable; --- Now Map table is ready and we can use it --- INSERT INTO [dbo].[tbl_StudentMarks] (StdRoll, SubjectName, MarksObtain) SELECT b.NewRecordID, a.StubjectName, a.MarksObtain FROM #tbl_TempStudrntMsrks AS a INNER JOIN @MappingTable AS b ON a.SrlNo = b.OldRecordID; END GO Step – 4 [ Observation ] SELECT * FROM [dbo].[tbl_StudentDetails]; GO SELECT * FROM [dbo].[tbl_StudentMarks]; GO StdRoll StudentName StudentClass StudentSection 100 Joydeep Das 1 A 101 Preeti Sharma 1 A 102 Deepasree Das 1 A IdNo StdRoll SubjectName MarksObtain 1 100 Bengali 50 2 100 English 70 3 100 Math 80 4 101 Bengali 0 5 101 English 70 6 101 Math 80 7 102 Bengali 20 8 102 English 90 9 102 Math 95

Written by Craig Wentworth. To understand the furor that’s greeted recent vendor announcements around open source analytics computing engine Spark, and some commentary seemingly setting up a Spark versus Hadoop battle, it’s worth taking a moment to recap on what each actually is (and is not). As I covered in last year’s MWD report on Hadoop and its family of tools, when people talk about Apache Hadoop they’re often referring to a whole framework of tools designed to facilitate distributed parallel processing of large datasets. That processing was traditionally confined to MapReduce batch jobs in Hadoop’s early days, though Hadoop 2 brought the YARN resource scheduler and opened up Hadoop to streaming, real-time querying and a wider array of analytical programming applications (beyond MapReduce). Spark has been designed to run on top of Hadoop’s Distributed File System (amongst other data platforms) as an alternative to MapReduce – tuned for real-time streaming data processing and fast interactive queries, and with multi-genre analytics applicability (machine learning, time series, graph, SQL, streaming out-of-the-box). It gets that speed advantage by caching in-memory (rather than writing interim results to disk, as MapReduce does), but with that approach comes a need for higher-spec physical machines (compared with MapReduce’s tolerance for commodity hardware). So, Spark isn’t about to replace Hadoop -- but it may well supplant MapReduce (especially in growing real-time use cases). Those “Spark vs Hadoop” headlines are about as meaningful as one proclaiming “mushrooms vs pizza." Yes, mushroom might be a more suitable topping than, say, pepperoni (especially in a vegetarian use case), but it’ll still be deployed on the same dough and tomato sauce pizza platform. Nobody’s about to suggest the mushroom should go it alone! But what’s behind the headlines and the hype is a story of enterprise adoption – or at least vendors anticipating that adoption and investing in ‘the weaponization of Spark’ as it faces the more exacting standards of security, scaling performance, consistency, etc. which come with mainstream enterprise deployment. Big names like IBM, Databricks (the company formed by the originators of Spark), and MapR made commitments in and around the Spark Summit earlier this month. MapR has announced three new Quick Start Solutions for its Hadoop distribution to help customers get started with Spark in real-time security log analytics, genome sequencing, and time series analytics; Databricks’ cloud-hosted Spark platform (formerly known as Databricks Cloud) has become generally available; and IBM announced a raft of measures designed to give Spark a significant shot in the arm – it’s open sourcing its SystemML technology to bolster Spark’s machine learning capabilities, integrating Spark into its own analytics platforms, investing in Spark training and education, committing 3,500 of its researchers and developers to work on Spark-related projects, and offering Spark as a service on its Bluemix developer cloud. Given the overlap with Databricks’ business model (of offering development, certification, and support for Spark), IBM’s intentions are likely to tread on some toes before long – but for now, at least, both companies are content to focus on the combined push benefiting the Spark community and its enterprise aspirations overall (though clearly IBM’s betting on all this investment buying it some influence over where Spark goes next). It’s worth bearing in mind that not all its supporters champion Spark wholesale and all the interested parties tend to be interested in particular bits of Spark (as wide-ranging as it is) because of overlaps with their own preferred toolsets. For instance, although Spark supports many analytics genres, Cloudera focuses on its machine learning capabilities (as it has its own SQL-on-Hadoop tool in Impala), and MapR and Hortonworks also promote Drill and Hive as their favoured source of SQL-on-Hadoop. IBM’s support is focused on Spark’s machine learning and in-memory capabilities (hence the SystemML open sourcing news). In the face of such strong vendor preferences, how long before some of Spark’s current features fall away (or at least start to show the effects of being starved of as much care and feeding as is bestowed upon vendors’ favourite Spark components)? The Spark community is at much the same place the Hadoop one was at a while back – it’s showing great promise and suitability in key growth workloads (in Spark’s case, such as real-time IoT applications). However, the product as it stands is too immature for many enterprise tastes. Cue enterprise software vendors stepping up to help grow Spark up fast. Their challenge though is to smooth out the edges without smothering what made it so interesting in the first place.

"Here is the specification; how much will it cost to create this software?" I hear this almost every day from clients of Teamed.io and prospects that are planning to become our clients and outsource the software development to us. My best answer is "I don't know; it depends." Sounds like a strange response for someone who claims he knows what he is doing, isn't it? "Here is the 20-page specification that explains all the features of the product; how come you can't estimate the cost?" I can, but I won't. Here is why. Let me ask you something else: Why do you need an estimate? Yes, I mean it — why do you ask me how much it will cost to develop the software for you? I can tell you why. Because you don't trust me. And obviously you have good reasons for that, simply because we both know that a software product is something that can stay in development forever and never be finished. Look at YouTube, for example. How much do you think it would take to create a website like this, where users are able to upload their videos and then stream them? A few days for a good web developer. Will it stream video? Yes, it will. Will it be ready to compete with YouTube? No, it won't. Add a few hundred developers to the team, a few years, and a few million dollars, and even then you will be behind YouTube. Simply because it's a never-ending process. Any software, no matter how big or good it is, always needs more and more improvements and bug fixes. Thus, when you ask me how much it will cost to create a system similar to YouTube, according to your specifications, my honest and accurate answer should be: "All of your money, and it won't be enough." Will you sign a contract and outsource the project to me after that answer? No, you won't. That's why I have to lie and promise something like "three months and $40,000". Why would you trust me? If you're smart enough, you won't. My point is that no matter what I promise you, I will be wrong. Terribly wrong. What is the solution? What do you do? I totally understand that you do need a number to make your plans and secure the money. You need to choose the right software outsourcing partner, and you need to know what to expect and when, but ... You're asking the wrong question! This question has only one valid answer, and it's the answer nobody will ever give you — the development will take forever and will consume all your money. All other answers are simply a lie. I'm sorry if I've delivered bad news to you. But let's get back to the original problem: Why are you asking me how much it will take to develop the software if you know it's a never-ending process and there is basically no limit? Because you want to make sure your $40,000 will be used the right way and will produce the maximum it can produce. To get this assurance from me, you're asking for an estimate. I'm telling you that your software will be ready for $40K, and you sleep well. Until the moment you realize you've been fooled. By your own self. Your concern is perfectly valid. You want to spend no more than $40K, and you want to get a product that will help you achieve your business goals. For example, you want to get into the market and acquire your first few thousand users. In other words, your biggest worry is that your dollars will be turned into the right amount of the right software. Any software team can consume your $40K, but each team will produce a different amount of software with different quality. My point is that instead of asking how much a software project will cost, you should ask how much software we can create for each dollar you give us and what quality will it be. Basically, don't ask us to estimate how much gas it will take to get to the finish line, because there is no finish line. Instead, ask us how much we charge per gallon and how many gallons we consume per mile. Different teams use different metrics to measure their results (to be honest, most of them don't use any). We, at Teamed.io, use hits of code, bugs, pull requests, test coverage, and a few other metrics as measurable indicators of quantity and quality. We know exactly how much software we can produce for each $100 you pay us. Collect those numbers from other teams and compare them. Also, make sure you can control these numbers during the course of the project. That's the guarantee you're looking for. Now you know what you're buying and how much you're paying for it. In other words, like I said above, having these numbers in front of you will assure you that your money is producing the maximum amount of software it can produce, at the highest quality. The only question left is how you can know you're buying the right software. In other words, you know how much we charge per gallon and how many gallons we use per mile, but how do you know we're driving in the right direction and not making too many circles or detours? There is only one mechanism to guarantee that: regular checkpoints. You should ask us whether we deliver the software in small and regular increments, and whether we allow you to conduct regular independent reviews of our progress. Also, make sure we prioritize our technical goals, use milestones, versionalize releases, publish release notes, etc. Make sure that in the course of our journey, you are able to control the progress and make corrections. To summarize, this is wrong (because there is no "there"): Hey driver, how much will it cost to get there? And this is right: Hey driver, how much do you charge per mile, and do you have a map? Hope I've made my point clear :)

Here is the third post in a six part series focusing on DevOps Days Austin. Following Cote’s presentation, next in the line up of speakers on day one was Paul Read of Release Engineering Approaches. Paul’s talk tackles the “importance of continually breaking your organization with experiments, some surprising examples, and how to do it in such a way that you aren’t left with a broken organization.” Take a listen, and check out his awesome tie! (also see his slides below) <br> Some of the ground Paul covers: Learning from Toyota Kata The culture of continuous learning and experimentation/improvement Making “the pain visible” Where Paul sees DevOps going in the future Still to come Cameron Haight – Gartner John Willis – Docker Barton George (Sputnik ignite talk) – Dell Extra credit reading The coming donkey apocalypse — Cote Opening Keynote – Damon Edwards Pau for now…

It’s fairly well established that nature can play a big part in our wellbeing and productivity. Indeed, I wrote about a study last year that revealed how a short walk through a park type environment can give our productivity and happiness a considerable boost. Could just hearing natural sounds trigger the same boost? That was the question posed by a recent study from academics at Rensselaer Polytechnic Institute. Their study suggests that simply hearing sounds, such as a flowing river, can improve both our mood and also our cognitive ability. Now, suffice to say, the modern open plan office is often a long way from anything remotely natural sounding, and indeed, many offices employ a masking system to try and limit the spread of noise throughout the office. Such systems enable you to listen effectively to those nearby you, but limits the spread of that chatter throughout the room. “If you’re close to someone, you can understand them. But once you move farther away, their speech is obscured by the masking signal,” the authors reveal. Noise in the workplace Now the presence of these things is news to me, but apparently they exist in most office ceilings and mask office chatter by outputting a form of electronic noise (or white noise if you like). The researchers wanted to test if they could use a form of masking signal that not only helps to block out the noise of open plan offices, but also helps to make us more productive. So rather than piping white noise into the office, they began to pipe in natural sounds instead. The hyptothesis they were testing was that people would focus more intently in such an environment, thus improving their productivity. The study saw participants exposed to a range of different sounds whilst they were undertaking a task that required an intense focus, which is just the kind of task that open offices are usually terrible at supporting. The noises included your usual office hubub, the white noise usually used to screen this noise, and a natural sounding masker, which in this case was a mountain stream. Nature for the win The results revealed that those exposed to the sound of the mountain stream were both more productive and generally happier than their peers who were exposed to more traditional office noises, both of the unmasked office din and the white noise based masking signal. Despite the relatively small sample used in the research, the authors are confident that they’re onto something, and that using such natural sounding maskers could have a big impact on the normal office environment, but also in environments such as a hospital where it could be used to improve the mental wellbeing of patients. It’s certainly an interesting finding and would potentially be a cheap, yet effective, way of making a substantial difference to our performance and wellbeing at work. Can you think of any other sounds that might have a similar effect as the mountain stream?

We’re going back to the classics this month, with the latest enterprise collaboration news round-up focussing on cloud strategy and considerations and benefits when it comes to implementing collaboration software. Mashable shared an infographic it created in conjunction with Hewlett Packard, which compiles data and research suggesting that the use of hybrid and private cloud computing is on the rise. The article quotes statistics from Rightscale, which states that 82% of enterprises have a multi-cloud strategy already, and of these 14% use multiple private clouds, 13% use multiple public clouds, and 55% use hybrid clouds. Mashable quotes Technology Business Research, which states that “there is continued migration of enterprise vendors in mature markets such as the U.S. to hybrid and private cloud platforms to provide software vendors an opportunity to generate adoption for management technologies, as customers require next-generation tools to manage heterogeneous IT infrastructures efficiently.” In his article for eWEEK, Chris Preimesberger outlines 10 ways IT and business leaders must collaborate on cloud strategies. Chris explains that a decision to use cloud services is no longer simply down to the IT department. During the last nine years, he says, entire businesses have become necessarily immersed in IT strategies in order to harness the cloud for economics, innovation, operations and growth. He shares a slide show which provides advice for how technical and business leaders can collaborate to build a secure cloud strategy. The slide show states that usage indicate that private clouds are expected to grow at double the rate of public cloud, a result of ongoing concerns about data security and privacy. Gary Audin asks the question cloud economics or flexibility? in his article for No Jitter. Gary explains that although the cost of cloud can be attractive, that might not be the real draw for enterprises. He states that knowing what costs to consider as part of a cloud service implementation is vital to making the right decision about cloud. Gary points out the benefits of the cloud as being far more than simply a matter of cost. He explains that the cloud allows rapid response for an enterprise as it contends with change due to situations such as staff growth or reduction, market fluctuations, financial limitations, or new opportunities. Above all, Gary explains, the cloud delivers flexibility and it is this which makes it the most attractive option for enterprises. In his article for MSP Mentor, Michael Brown reveals the result of a recent report on cloud adoption in the enterprise. The report, by Skyhigh Networks, revealed that enterprise cloud adoption grew by 43% in 2014. Michael highlights findings on the file sharing front, revealing that 37 percent of employees were found to be uploading sensitive business data to consumer file sharing services. Consumer file sharing services are one element of a growing trend towards BYOC (bring your own cloud, content and collaboration). Robert Bamforth explains that BYOC is an evolution of BYOD (bring your own device) which posed a challenge to IT departments since the rise of the smartphone. Robert explains that BYOC is a new challenge for IT departments in controlling their organisation’s digital assets while liberating employee productivity and information sharing. Robert states that the BYOC conundrum should change as enterprise-strength security features and tools continue to evolve to have more consumer-like interfaces, which will make asking employees to use enterprise tools much easier. He gives some suggestions to help enterprises in the mean time: understand the appeal of consumer tools, make sure everyone understands security risks, forget trying to apply strong rules to trivial information, get a mobile-ready solution, look for and pre-plug data leaks, and above all don’t stop collaboration if it’s happening. In his article for ZDNet, Dion Hinchcliffe reflects on the state of the digital collaboration industry. Far from maturing, Dion says, the collaboration tool space is busier than ever evolving, branching out, and multiplying. But, he asks, are organizations able to adopt so many different ways of working together? Dion observes that instead of settling down, the collaboration software space is actually get more interesting and varied, and he is seeing new technologies, such as applications that focus on optimizing collaboration for mobile devices or for team analytics. It’s now time for organizations to design a strong foundation for digital collaboration, says Dion, as the near future promises many key new innovations that must be considered and incorporated to stay competitive, both to customers and the workforce. When businesses do decide to adopt one or more digital collaboration platforms, Andre Bourque offers some helpful ways in which to measure ROI. Andre quotes a Mashable report which states that cloud collaboration drives creativity and engagement, leading to happier employees and a better company culture, but this is not a metric that is easily measurable. Andre explains that it’s hard to find definitive examples of ROI, as most are anecdotal or “in process”, and merely counting user adoption rate of a collaborative platform is inadequate. Instead, Andre quotes Angela Ashenden, of MWD Advisors, who offers the following metrics to consider: reduced travel time and costs; creating new business opportunities and services; increased employee retention rates, cost savings across the organisation, and faster on-boarding for new users. Do you have any metrics that you find useful to measure ROI on your collaboration platform in your organisation?

Docker deployments can be very dynamic with containers being started and stopped, moved around the YARN or Mesos-managed clusters, having very short life spans (the so-called pets) or long uptimes (aka cattle). Getting insight into the current and historical state of such clusters goes beyond collecting container performance metrics and sending alert notifications. If a container dies or gets paused, for example, you may want to know about it, right? Or maybe you’d want to be able to see that a container went belly up in retrospect when troubleshooting, wouldn’t you? Just two weeks ago we added Docker Monitoring (docker image is right here for your pulling pleasure) to SPM. We didn’t stop there — we’ve now expanded SPM’s Docker support by adding Docker Event collection, charting, and correlation. Every time a container is created or destroyed, started, stopped, or when it dies, spm-agent-docker captures the appropriate event so you can later see what happened where and when, correlate it with metrics, alerts, anomalies — all of which are captured in SPM — or with any other information you have at your disposal. The functionality and the value this brings should be pretty obvious from the annotated screenshot below. Like this post? Please tweet about Docker Events and Docker Metrics Monitoring Know somebody who’d find this post useful? Please let them know… Here’s the list of Docker events SPM Docker monitoring agent currently captures: Version Information on Startup: server-info – created by spm-agent framework with node.js and OS version info on startup docker-info – Docker Version, API Version, Kernel Version on startup Docker Status Events: Container Lifecycle Events like create, exec_create, destroy, export Container Runtime Events like die, exec_start, kill, oom, pause, restart, start, stop, unpause Every time a Docker container emits one of these events spm-agent-docker will capture it in real-time, ship it over to SPM, and you’ll be able to see it as shown in the above screenshot. Oh, and if you’re running CoreOS, you may also want to see how to index CoreOS logs into ELK/Logsene. Why? Because then you can have not only metrics and container events in one place, but also all container and application logs, too! If you’re using Docker, we hope you find this useful! Anything else you’d like us to add to SPM (for Docker or anyother integration)? Leave a comment, ping @sematext, or send us email – tell us what you’d like to get for early Christmas!

DevOps can help reduce technical debt in some fundamental ways. Continuous Delivery/Deployment First, building a Continuous Delivery/Deployment pipeline, automating the work of migration and deployment, will force you to clean up inconsistencies and holes in configuration and code deployment, and inconsistencies between development, test and production environments. And automated Continuous Delivery and Infrastructure as Code gets rid of dangerous one-of-a-kind snowflakes and configuration drift caused by making configuration changes and applying patches manually over time. Which makes systems easier to setup and manage, and reduces the risk of an un-patched system becoming the target of a security attack or the cause of an operational problem. A CD pipeline also makes it easier, cheaper and faster to pay down other kinds of technical debt. With Continuous Delivery/Deployment, you can test and push out patches and refactoring changes and platform upgrades faster and with more confidence. Positive Feedback The Lean feedback cycle and Just-in-Time prioritization in DevOps ensures that you’re working on whatever is most important to the business. This means that bugs and usability issues and security vulnerabilities don’t have to wait until after the next feature release to get fixed. Instead, problems that impact operations or the users will get fixed immediately. Teams that do Blameless Post-Mortems and Root Cause(s) Analysis when problems come up will go even further, and fix problems at the source and improve in fundamental and important ways. But there’s a negative side to DevOps that can add to technical debt costs. Erosive Change Michael Feathers’ research has shown that constant, iterative change is erosive: the same code gets changed over and over, the same classes and methods become bloated (because it is naturally easier to add code to an existing method or a method to an existing class), structure breaks down and the design is eventually lost. DevOps can make this even worse. DevOps and Continuous Delivery/Deployment involves pushing out lots of small changes, running experiments and iteratively tuning features and the user experience based on continuous feedback from production use. Many DevOps teams work directly on the code mainline, “branching in code” to “dark launch” code changes, while code is still being developed, using conditional logic and flags to skip over sections of code at run-time. This can make the code hard to understand, and potentially dangerous: if a feature toggle is turned on before the code is ready, bad things can happen. Feature flags are also used to run A/B experiments and control risk on release, by rolling out a change incrementally to a few users to start. But the longer that feature flags are left in the code, the harder it is to understand and change. There is a lot of housekeeping that needs to be done in DevOps: upgrading the CD pipeline and making sure that all of the tests are working; maintaining Puppet or Chef (or whatever configuration management tool you are using) recipes; disciplined, day-to-day refactoring; keeping track of features and options and cleaning them up when they are no longer needed, getting rid of dead code and trying to keep the code as simple as possible. Microservices and Technology Choices Microservices are a popular architectural approach for DevOps teams. This is because loosely-coupled Microservices are easier for individual teams to independently deploy, change, refactor or even replace. And a Microservices-based approach provides developers with more freedom when deciding on language or technology stack: teams don’t necessarily have to work the same way, they can choose the right tool for the job, as long as they support an API contract for the rest of the system. In the short term there are obvious advantages to giving teams more freedom in making technology choices. They can deliver code faster, quickly try out prototypes, and teams get a chance to experiment and learn about different technologies and languages. But Microservices “are not a free lunch”. As you add more services, system testing costs and complexity increase. Debugging and problem solving gets harder. And as more teams choose different languages and frameworks, it’s harder to track vulnerabilities, harder to operate, and harder for people to switch between teams. Code gets duplicated because teams want to minimize coupling and it is difficult or impossible to share libraries in a polyglot environment. Data is often duplicated between services for the same reason, and data inconsistencies creep in over time. Negative Feedback There is a potentially negative side to the Lean delivery feedback cycle too. Constantly responding to production feedback, always working on what’s most immediately important to the organization, doesn’t leave much space or time to consider bigger, longer-term technical issues, and to work on paying off deeper architectural and technical design debt that result from poor early decisions or incorrect assumptions. Smaller, more immediate problems get fixed fast in DevOps. Bugs that matter to operations and the users can get fixed right away instead of waiting until all the features are done, and patches and upgrades to the run-time can be pushed out more often. Which means that you can pay off a lot of debt before costs start to compound. But behind-the-scenes, strategic debt will continue to add up. Nothing’s broke, so you don’t have to fix anything right away. And you can’t refactor your way out of it either, at least not easily. So you end up living with a poor design or an aging technology platform, slowly slowing down your ability to respond to changes, to come up with new solutions. Or forcing you to continue filling in security holes as they come up, or scrambling to scale as load increases. DevOps can reduce technical debt. But only if you work in a highly disciplined way. And only if you raise your head up from tactical optimization to deal with bigger, more strategic issues before they become real problems.

LastPass, the password manager that lets you manage your passwords between different devices, was recently hacked. From this there has been a fair amount of FUD circulating and not enough rational thought. With that in mind, this seems like a good time to talk about password security and LastPass with some rational ideas. Since we can't get rid of passwords just yet we need to manage them well. 1. LastPass Detected The Breach No useful system is impenetrable. Computers not connected to the Internet, that don't even have a network card, have been hacked across an air gap using their speakers and mic. The most up to date systems still suffer from zero-day exploits. Two of the elements of an organization that takes security seriously are keeping certain pieces of data separate and detecting when a breach occurs. From the LastPass announcement of the breach we can see these two things in action. While some information was obtained the actual vaults of passwords were not downloaded. And, they detected there was a problem and enough monitoring in place to distinguish what was effected. I can't overstate how nice that is to hear. Many organizations won't detect if they have been breached. Even many of those that could detect a breach wouldn't be able to tell you what was affected. That's right, many of the places you put personal information couldn't do what LastPass did. 2. LastPass Responded To The Breach Even though the password vaults were not taken LastPass is having everyone change their vault password. They detected the problem and are going the extra mile to protect their users. Now, let's consider an alternative option. Consider a 1Password or KeePass user who stores their information in Dropbox or a similar service. A malicious program on one of their systems could have taken their vault and sent it to an attacker. Those users would not have known. Or, the service could have been hacked but since it's not password specific who would have suggested changing the master password? I'm not trying to defend LastPass. It's a matter of considering the alternatives and the security measures around them. Is a 1Password or KeePass alternative setup actually more secure in practice? 3. Different Passwords For Different Sites There's a good reason to have a different password or passphrase for different sites. You can't trust that a site you submit it to will store it securely so that it won't be misused to access other sites. With all the sites we connect to we it's difficult to remember a different password for each site. Congratulations if you can do that. For the rest of us we need a system to help. This is where a password manager is useful. That is, until we can stop using passwords for something better. So, use a password manager if you can't otherwise have a different password for each site. It's more secure than using the same password everywhere. Note, I'm not recommending a particular password manager on purpose. Use a good one. 4. Security Is Not About Perfection There is no such thing as perfect security. Security needs to be practical. For example, for most people it's more security to use a password manager than to use the same password everywhere. Neither is perfect but when you weigh the differences the password manager comes out as more secure. In a distributed device world where we need passwords on more than one system it's good to go with a system that does this for you. A system that focuses on security and handling issues that come up. You could roll your own solution. But, will it be more secure? For most people the professional solution is the more secure one. When considering password security choose the one that's more secure for you rather than seeking the perfect option. 5. When Not To Use A Password Manager There are some places I would recommend not using a password manager. For example, I would recommend not using one for your financial sites. Those few places that are very important use a passphrase. 6. Encrypt Your Password Store In our multi-device world you'll likely need to share your password datastore between devices. And, any device can be hacked even if you don't need to use multiple devices. Imagine a virus on a computer looking for your password excel file and uploading that to someone bad. It happens. Use an encrypted datastore. This is why password managers are important. They are designed to store your data in an encrypted manner. This way, if someone gets your data store they will have a very hard time reading it. Before they can get to anything they'll need to break the encryption which isn't so easy. That means, even if an attacker had gotten the password vaults from LastPass, which they didn't, they would not have been able to read the data in them. 7. Limit Your Attack Vector One of the problems with LastPass is that they are a known password manager. That makes them a target if someone wants to try and get passwords. Alternatives that store their distributed information in general purpose systems pose a different attack vector. For example, if you use 1Password or KeePass and store your information in Dropbox you can still be hacked. Dropbox has been externally hacked in the past and other applications can access your Dropbox folder. Using alternatives to LastPass doesn't mean you won't be attacked. Take a few minutes and consider the attack vectors of the different solutions you're considering and how each of those will detect a breach and respond to that. For example, I could self host my encrypted file on the Internet somewhere. This would be managed by me and wouldn't be a known system for someone to target. But, the IPv4 addresses are regularly checked for known vulnerabilities so attackers and get onto a system a poke around. That is the entire IPv4 space, which is still the only space routable for all things, is regular checked. Will I keep everything on that system up to date? Will I detect if someone broke into the system? Will I respond appropriately? All of this needs to be taken into account. Final Thoughts On LastPass I'm not trying to defend LastPass. I'm trying to give a little more of a holistic picture of security. It's complicated and any alternatives to a LastPass or LastPass-like solutions need to have their security considered. Viewing the options with security and attacks in mind keeps everything in perspective.

Business leaders like to be in control of every aspect of their operations, but if any element is outsourced, that sense of governance becomes much more difficult to maintain. Outsourcing can be appealing to organizations looking for talent at reasonable costs - it just takes significant planning to pull off successfully. When a team is in a different location from the company, there are a number of challenges that must be overcome. We will look at what some of the biggest issues are and how you can appropriately manage an outsourced quality assurance team. Obstacles of outsourcing While outsourcing can have a lot of advantages for businesses, the number of roadblocks can be intimidating for many enterprises. Here are the biggest challenges to prepare for: Information sharing: When you don't see individuals every day, it can often be easy to forget to relay important messages. Communication in this situation is imperative, as it could affect the overall operation of the application. As soon as a request is given, there should be seamless transfer of knowledge to ensure that everyone is on the same page and that the project proceeds as expected. This will eliminate any redundancies and lower the overall development cost. Engagement: Many outsourced teams have a difficult time becoming personally involved with their projects. This will also affect their ability to collaborate effectively with other teams in the business, ultimately hurting program quality. Organizations must have a strategy to keep these individuals motivated and provide them with the tools to succeed. Technology/skills: An outsourced team may use different pieces of technology or have skills separate from what the organization was looking for. For example, if the company really wants to move to agile software development, but the outsourced group still uses waterfall methods, that could create problems down the road for their software development initiatives. Similarly, the business must ensure that the outsourced individuals have the skills necessary to meet corporate goals and spur innovation through their app testing. How to regain control Although total governance of outsourced assets won't be possible, there are still some things that organizations can do to take control of these teams and ensure that they're fulfilling business objectives. Australian outsourcer Beepo suggested a consistent schedule for gathering feedback and using technology like the cloud and test management tools. Let's dissect each of these ideas. Many outsourced teams are often left by the wayside when it comes to communication. This can reasonably lead to mistakes being made and leave the members feeling apathetic toward their work. However, by setting up regular video conferences and requesting feedback, outsourced individuals can feel empowered to express their opinions and become a larger part of their projects. This will also help build trust and motivate teams to collaborate more. Using tools like test management and the cloud can also be helpful when working with an outsourced team due to the fact that they provide a singular platform for all users. This means that the outsourced and in-house teams can be working on the same project at the same time, with any changes being made in real time. This will not only reduce redundancies, but it creates accountability and ensures that tasks are being addressed according to their priority. Considerations to make when outsourcing Whether you're looking to outsource, or simply make your outsourced team better, there are some key items to address. Ashok Mani from AppLabs noted that organizations must look into a provider's engagement models, mobilization efforts, communication plans, security and scalability. These elements will be essential to clear up before trying to manage a team. "While organizations are deriving value from outsourcing software development, outsourced software testing will maximize returns from their investments and provide the right level of objectivity and rigor required to create a high-quality product," Mani stated. "If an independent QA and testing service provider is chosen whose focus is on ensuring quality products/systems are implemented, benefits will be fully maximized." Outsourcing a quality assurance team is going to have a few challenges for businesses. But by preparing for these obstacles, they will be able to manage the outsourced group more effectively. Having a communication plan and technology available will be essential to working well with the team and improving development operations.

If you are building mobile apps then you are connecting to some REST API. For example, if you want to resolve an address to a latitude/longitude information to display on a map, you might use the Google Geocoding API: Google Geocoding API: https://maps.googleapis.com/maps/api/geocode/json?address=San Francisco,CA&key=AIzaSyDvFMYGjeR02RH If you are invoking the API from the client, then the API key also has to be present on the client. But, this is also the problem. It’s very easy to look at the app source in the browser and get access to the API key. If someone has access to your API key, they can send requests on your behalf (without you knowing), and use up your request quota. Even if you are building a hybrid app, it’s still the same problem. A hybrid app is HTML/JavaScript inside a native wrapper, it’s possible to download the app, un-package it and gain access to API keys or any sensitive information stored in the app. Even native apps are not immune to this. For example, an Android app is just a Java application and a Java application can be de-compiled to view the original source. The next image shows how to get access to an API key in the browser: Viewing app source in browser A good solution is to never expose the API key (or any other sensitive data) on the client. How do you do that? You keep the API key and any other sensitive information on the server. Appery.io Secure Proxy Appery.io Secure Proxy (part of Backend Services) enables app developers to keep sensitive app data on the server. Your API keys or any other data is never exposed on the client. Watch this 5-minute video on how to use Secure Proxy: Before using the Secure Proxy, you need to store the data on the server. To store the data you are going to use the Appery.io Database. It’s as simple as creating a collection with two columns. The first column is the value name, the second column is the actual value. This is how the database looks when storing the API key for Google Geocoding API: Saving API key in database As this key is stored on the server, no one (but you) has access to it. You can store other data as well such as URLs, tokens or anything else that shouldn’t be exposed on the client. The next step is to setup the proxy that will use the information stored in the database. This step is also very simple, this is how it looks: Secure proxy linked to a database You give the proxy a name and then link it to a database which stores your data. The above proxy is linked to Secrets_db database, Credentials collection, and secretName, secretValue columns. The last step is to link a REST API service to the proxy. In the service editor you select the secure proxy created: REST API service using secure proxy then in the Request tab you reference the API key stored in the database (the name stored in secretName column): Request parameter substitution will happen on the server and that’s it. When the API service is invoked, the call will go through the secure proxy (server) where the API key will substituted: API key is not exposed on the client For web apps, you can add an extra layer of security by specifying from which page URLs the proxy should accept requests: URL-based security The proxy will only accept requests from page URLs listed in the table. Another option to keep API keys private is to invoke the API from the server using Server Code, I will cover this in another post. Setting up an using the Appery.io Secure Proxy is simple. It provides a very important feature by allowing to keep sensitive and private data on the server, never exposing it on the client, and adding an extra security layer to your app.

I am sure by now most people understand that there is strong correlation between employee satisfaction and business results. If you need more convincing have a read of these two articles: Forbes & Research Paper So how do you best go about measuring it? On my current project I have decided to go with the following 4 questions: I would recommend this account and my project as a good place to work I have the tools and resources to do my role well I rarely think about rolling off this account or project My role makes good use of my skills and abilities For those of you who have read Jez Humble’s “Lean Enterprise”, these questions will look familiar. I have adopted them to the project setting that I work within. We have just set out on a cultural transformation to become truly Agile and adopt DevOps in a large complex legacy environment. To me measuring the above will give me the best indicator that we are doing the right thing. Of course there will be other measures who determine the quality of the outcomes and the levels of automation among others, but changing the culture of an organisation is critical if your Agile and DevOps adoption is to be successful. I will report back throughout that journey to tell you what my experiences is with the above questions. IT delivery is complex and it is not always clear what the right solution is. I found in the past that it is near impossible to create processes and tools that work by itself, you need to have the right mindset that people use the processes and tools with the right intent. It’s very frustrating when you implement great automation only to see a few months later that the solution has degraded. It is with hindsight that I understand that the solution is to not just implement process and tools but to instill the right culture and mindset for progression, a culture where we blamelessly identify a way to avoid the same mistake again rather than looking for the person in fault, a culture where we strive for automation and lean processes and are not concerned about the size of our teams or budgets, a culture where you don’t have to protect your fiefdom and where you are happy to collaborate with others to solve problems no matter where the root cause lies. I think we all in IT need to understand this dynamic between employee satisfaction and outcomes better, I for sure believe that I have come across a magic potion that I aim to bring to all my future projects. About these ads