One of the most important kinds of data a Platform as a Service (PaaS) can leverage is its knowledge of guest applications that run within its purview. A PaaS should know all sorts of things about guest applications – their architecture, dependencies, scale across infrastructure, and more. Data including application resource utilization metrics (CPU, RAM, etc.) are key for things like data center capacity planning, policy enforcement, and application isolation in the enterprise. A PaaS such as Apprenda provides this information through a centralized single lens – in our case, a collection of RESTful APIs – making it easier than ever before to run analytics on application metrics in the data center. Apprenda’s approach as a PaaS is to provide developers and platform operators with helpful information through platform extensibility and APIs. This is because there are plenty of tools in the data center that provide advanced analytic capabilities, so long as you can feed them the information they need. We integrate with tools like System Center, New Relic, and more all the time because these are the tools our customers have invested in, and they are great at what they do. Our job is not to reinvent these tools but instead to provide data. Apprenda captures information about applications such as their duration of deployment, resource policy (allocation of CPU and memory), actual utilization of resources, scale (# of instances), custom metadata, and more. All of this information can be fed into data center tools that help IT make important, data-driven, decisions. In the land of DevOps, however, it is not uncommon for folks to use this data in creative and innovative ways. Often times this means using the mechanism “du jour,” which can be scripting (PowerShell), a programming language (R), or an entire runtime (Node.js) to quickly and effectively grab, process, and manipulate data. In a big example, let’s look at R, which is a powerful programming language centered on data mining and statistical analysis. It provides straightforward facilities for many types of data-analytics techniques, and is extensible using community maintained packages. In the simple example below, I use standard R functions plus three packages (easily included using R’s install.packages() function): 1. jsonlite for parsing JSON data that the Apprenda API returns. 2. httr for handling the HTTP requests necessary to authenticate and retrieve data. 3. plotrix for help rendering a plot of retrieved data. From there it’s pretty straightforward. The first step is to authenticate with your Apprenda environment: I’ve now stored my Apprenda session token in a variable called ‘token.’ I’ll include that token as a header in my API call to get application data: GET() is a function provided by the httr package that simplifies an HTTP request to the API. I’ve added the Apprenda session token to the HTTP Headers for authentication, and included a query string parameter that will help return all currently running application workloads on the platform. The data that is returned is parsed and stored in the variable (in R, a vector) called ‘r’ which now has 151 records, one for each application workload. Each record in ‘r’ has 15 variables (properties) that we can use to run analytics across the entire collection of results. For the purposes of illustration, I’m going to use the variable componentType, which represents Apprenda’s knowledge of the type of application workload that was deployed – there are seven self-explanatory types: UserInterface, PublicUserInterface,WindowsService, JavaWebApplication, LinuxService, WcfService, and Database. When the collection is then grouped bycomponentType, it becomes pretty simple to plot a chart showing the distribution of workload by the type of component: The resulting plot (pie3D() comes from the plotrix package) looks like this: I’ve had conversations with IT folks who couldn’t describe the architectural makeup of their application portfolio in any level of detail, yet in this case we pulled the data in real time with one line of R. Admittedly, a pie chart is a pretty watered down way to look at this information, but the point is the data is available and can be grouped, filtered, manipulated, and analyzed very simply with R. For this example, I used the open-source edition of RStudio. Some other powerful information that could be gleaned from the platform’s APIs: 1. The average discrepancy between resource allocation and actual utilization per workload. (This is helpful in capacity planning.) 2. The longest -running application workload. 3. The most distributed applications. (This could aid in scaling decisions.) There are many more. A PaaS such as Apprenda is, by nature, in a unique spot in the data center stack because it maintains knowledge of both infrastructure and applications. It also serves as a hub for data that, when analyzed creatively, provides new insights. These insights are an opportunity for enterprises to enhance their practices to better serve developers and applications while operating more efficiently than ever.

Every business has unique characteristics that set it apart from other organizations, even within the same industry. For this reason, it shouldn't be surprising that there's not a one-size-fits-all approach to app security testing. Each company has certain protection expectations and regulations to adhere to, making it essential to find the best way to achieve these goals. Here are a few examples of security testing methodologies available for quality assurance teams to leverage: Black box With black box testing, QA professionals put themselves in the shoes of the hacker and attempt to break the app through various attack vectors. This methodology can yield a lot of information and help better secure the program from actual threats. A white paper by Security Innovation noted that software testers first analyze the system's architecture and business model to identify any security vulnerabilities. Looking over the software logic in this way can uncover subtle security and privacy issues that may not have been noticed otherwise, such as defects in design, input, system dependency, authentication, cryptography and information disclosure. "Although white box code inspection is good for analyzing static behavior, only black box exploratory testing can determine the dynamic behavior of how a system is implemented and used, the coupling between systems and the interactions of the distributed systems," Security Innovation wrote. Dynamic For QA teams that like to execute code, dynamic testing is the approach for them. This methodology checks the running application for how it behaves and responds to a variety of inputs. This is done to ensure that the product meets up with established regulations and is giving the expected outcomes. IBM noted that dynamic analysis is especially useful to identify code coverage, as it can discover bugs in paths that have gone untested. While dynamic testing can be manual work for testers, it can also yield significant information that will help mitigate defects and produce quality products. Static In contrast to dynamic testing, static approaches directly review the source code, often through an automated test management solution. TechTarget contributor Michael Cobb noted that this methodology occurs at the implementation phase, rather than when the app is running, and often helps mitigate vulnerabilities involved with industry compliance standards. Automation in this area can reduce the amount of time it takes to complete these tasks. However, it may not be able to detect sophisticated threats, which can be supplemented by dynamic security testing. "A thorough source code review has an advantage over dynamic testing," Cobb wrote. "Nothing is hidden from analysts during a source code review, so they can examine exactly how data flows through a program. By solving the problem at the code level, static testing reduces the number of security-related design and coding defects, and the severity of any defects that make it through to the release version, thus dramatically improving the overall security of the application." There are a number of security testing methodologies that organizations can pursue, and designing a combination of approaches may result with a solution that's best for their needs. Using these strategies, companies can better ensure the protection of sensitive information while providing users with the software testing tools needed to succeed.

Troy Hunt (@troyhunt, blog) had a great, very hands-on 2-day workshop about webapp security at NDC Oslo. Here are my notes. Highlights – resources Personal security and privacy https://www.entropay.com/ – a Prepaid Virtual Visa Card mailinator.com – tmp email f-secure VPN https://www.netsparker.com/ – scan a site for issues (insecure cookies, framework disclosure, SQL injection, …) (lot of $k) Site security https://report-uri.io/ – get reports when CSP rules violated; also displays CSP headers for a site in a human-friendly way https://securityheaders.io/ check quality of headers wrt security free SSL – http://www.startssl.com/, https://www.cloudflare.com/ (also provides web app firewall and other protections) ; SSL quality check: https://www.ssllabs.com/ssltest/ https://letsencrypt.org/ – free, automated, open Certificate Authority (Linux Found., Mozilla) Breaches etc. http://arstechnica.com/security/2015/06/hack-of-cloud-based-lastpass-exposes-encrypted-master-passwords/ https://twitter.com/jmgosney – one of ppl behind http://passwordscon.org . http://password-hashing.net experts panel. Team Hashcat. http://arstechnica.com/security/2012/12/25-gpu-cluster-cracks-every-standard-windows-password-in-6-hours/ To follow ! http://krebsonsecurity.com/ ! http://www.troyhunt.com/ ! https://www.schneier.com/ ! https://twitter.com/mikko (of F-Secure) also great [TED] talks kevin mitnick (jailed for hacking; twitter, books) Books http://www.amazon.com/We-Are-Anonymous-LulzSec-Insurgency/dp/0316213527 – easy read, hard to put down http://www.amazon.com/Ghost-Wires-Adventures-Worlds-Wanted/dp/1441793755 – about Mitnick’s hacking, social engineering, living on the run ? http://www.amazon.com/Art-Intrusion-Exploits-Intruders-Deceivers/dp/0471782661/ Mitnick: http://www.amazon.com/Art-Deception-Controlling-Element-Security/dp/076454280X/ – social engineering Other https://www.xssposed.org/ See https://www.drupal.org/SA-CORE-2014-005 https://www.youtube.com/watch?v=Qvhdz8yE_po – Havij example http://www.troyhunt.com/2013/07/everything-you-wanted-to-know-about-sql.html, http://www.troyhunt.com/2010/05/owasp-top-10-for-net-developers-part-1.html, http://www.troyhunt.com/2012/12/stored-procedures-and-orms-wont-save.html, Googlee: find config files with SA access info: `inurl:ftp inurl:web.config filetype:config sa` https://scotthelme.co.uk/hardening-your-http-response-headers/ and https://securityheaders.io/ https://developer.mozilla.org/en-US/docs/Web/Security/Public_Key_Pinning – prevent MITM wappalyzer chrome plugin displaying info about the server and client that can be detected (jQuery, NewRelic, IIS, win OS, …) http://www.troyhunt.com/2015/05/do-you-really-want-bank-grade-security.html http://www.troyhunt.com/2012/05/everything-you-ever-wanted-to-know.html tool: https://github.com/gentilkiwi/mimikatz extract plaintexts passwords, hash, PIN code and kerberos tickets from memory on Windows Notes HackYourselfFirst.troyhunt.com – an example app with many vulnerabilities Note: maximizing your browser window will share info about your screen size, which might help to identify you haveibeenpwned.com – Troy’s online DB of hacked accounts Tips check robots.txt to know what to access Example Issues no https on login page insecure psw requirements cookies not secure flag => sent over http incl. AuthCookie) psw sent in clear text in confirm email user enumeration, f.eks. an issue with AdultFriendFinder – entry someone’s email to login to find out whether they’ve an account post illegal chars, get them displayed => injection no anti-automation (captcha) login confirm. email & autom. creating 1m accounts => sending 1m emails => pisses ppl off, likely increase one’s spam reputation (=> harder to send emails) brute-force protection? ### XSS Reflected XSS: display unescaped user input Encoding context: HTML, JS, CSS … have diff. escape sequences for the same char (e.g. <) – look at where they’re mixed Check the encoding consistency – manual encoding, omitting some chars JS => load ext resources, access cookies, manipulate the DOM Task: stal authCookie via search ### SQL injection Error-based injection: when the DB helps us by telling us what is wrong -> use ti learn more and even show some data Ex.: http://hackyourselffirst.troyhunt.com/Make/10?orderby=supercarid <—— supercarid is a column name orderby=(select * from userprofile) … learn about DB sructure, force an exception that shows the valueex.: (select top 1 cast(password) as int from userprofile) => “Conversion failed for the nvar value ‘passw0rd …’” Tips think of SQL commands that disclose structure: sys.(tables,columns), system commands enumerate records: nest queries: select top X ows asc then top 1 rows from that desc write out how you think the query works / is being constructed internally cast things to invalid types to disclose values in err msgs (or implicit cast due to -1 ..) #### Defenses whitelist input data types (id=123 => onlyallow ints) enumerable values – check against an appropr. whitelist if the value is stored – who uses it, how? making query/insertion safe permissions: give read-only permissions as much as possible; don’t use admin user from your webapp ### Mobile apps Look at HTTP req for sensitive data – creds, account, … Apps may ignore certificate validations In your app: param tampering, auth bypass, direct object refs Weak often: airlines, small scale shops, fast foods, … Tips certificate pining – the app has the fingerprint of the server cert. hardcoded and doesn’t trust even “valid” MITM certificate (banks, dropbox, …)x ### CSRF Cross-Site Request Forgery = make the user send a request => their auth cookie included async Ajax req to another site forbidden but that doesn’t apply to normal post Protection anti-forgery tags ### Understanding fwrk disclosure http://www.shodanhq.com/ -> search for “drupal 7” -> pwn How disclosed: headers familiar signs – jsessionid cookie for java, … The default error and 404 responses may help to recognize the fwr HTML code (reactid), “.do” for Sttruts implicit: order of headers (Apache x IIS), paths (capitalized?), response to improper HTTP version/protocol, => likely still possible to figure out the stack but not possible to simple search for fwrk+version ### Session hijacking Steal authentication cookie => use for illegal requests. Persistence over HTTP of auth., session: cookie, URL (but URL insecure – can be shared) Session/auth ID retrieval: insecure transport, referrer, stored in exceptions, XSS Factors limiting hijacking: short duration expiry, keyed to client device / IP (but IPs may rotate, esp, on mobile devices => be very cautious) DAY 2 ——– ### Cracking passwords Password hashing: salt: so that 2 ppl choosing the same psw will have a different hash => cracking is # salts * # passwords inst. of just N has cracking tips: character space Dictionary: passw0rd, … Mutations: manipulation and subst. of characters Tips: 1Password , LastPass, …. GPU ~ 100* faster than CPU #### Ex: Crack with hashcat common psw dict + md5-hashed passwords => crack ./hashcat-cli64.bin –hash-type=0 StratforHashes.txt hashkiller.com.dic # 23M psw dict -> Recovered.: 44 326/860 160 hashes [obs duplications] in 4 min (speed 135.35k plains) Q: What dictionary we use? Do we apply any mutations to it? ### Account enumeration = Does XY have an account? Multiple vectors (psw reset, register a new user with the same e-mail, …) Anti-automation: is there any? It may be inconsistent across vectors Does it matter? (<> privacy needs) How to “ask” the site and how to identify + and – responses? Timing attacks: distinguish positive x negative response based on the latency differing between the two ### HTTPS Confidentiality, Integrity, Authenticity Traffic hijacking: [a href="https://www.wifipineapple.com/"]https://www.wifipineapple.com/ – wifi hotspot with evil capabilities monitor probe requests (the phone looks for networks it knows), present yourself as one of those, the phone connects autom. (if no encryption) Consider everything sent over HTTP to be compromised Look at HTTPS content embedded in untrusted pages (iframes, links) – e.g. payment page embedded in http Links HSTS Preload – tell Chrome, FF that your site should only be ever loaded over HTTPS – https://hstspreload.appspot.com/ https://www.owasp.org/index.php/HTTP_Strict_Transport_Security header ### Content Scurity Policy header https://developer.chrome.com/extensions/contentSecurityPolicy See e.g. https://haveibeenpwned.com/ headers w/o CSP anything can be added to the page via a reflected XSS risk Anyth, can be added to the DOM downstream (on a proxy) With CSP the browser will only load resources you white-list; any violations can be reported Use e.g. https://report-uri.io/home/generate to create it and the report to watch for violations to fine tune it. ### SQL injection cont’d (Yesterday: Error-Based) #### Union Based SQLi Modify the query to union whatever other data and show them. More data faster than error-based inj. Ex.: http://hackyourselffirst.troyhunt.com/CarsByCylinders?Cylinders=V12 : V12 -> `V12′ union select voteid, comments collate SQL_Latin1_General_CP1_CI_AS from vote– ` #### Blind Boolean (laborious) Blind inj.: We can’t always rely on data being explicitly returned to the UI => ask a question, draw a conclusion about the data. Ex: http://hackyourselffirst.troyhunt.com/Supercar/Leaderboard?orderBy=PowerKw&asc=false -> ordedby => case when (select count(*) from userprofile) > 1 then powerkw else topspeedkm end Extract email: Is ascii of the lowercase char #1 < ascii of m ? Automation: SqlMap #### Time based blind injection When no useful output returned but yes/no responses differ significantly in how much time they take. F.ex. ask the db to delay the OK response. MS SQL: IF ‘b’ > ‘a’ WAITFOR DELAY ’00:00:05′ ### Brute force attacks Are there any defences? Often not How are defences impl? block the req resources block the src IP rate limit (by src IP) ### Automation penetration testing apps and services such as Netsparker, WhiteHatSec targets identification: shodan, googledorks, randowm crawling think aout the actions that adhere to a pattern – sql injection, fuzzing (repeat a req. trying diff. values for fields – SQLi, …), directory enumeration automation can be used for good – test your site tip: have autom. penetration testing (and perhaps static code analysis) as a part fo your build pipeline Task: Get DB schema using sqlmap (see python2.7 sqlmap.py –help) ### Protection Intrusion Detection System (IDS) – e.g. Snort Web Application Firewall (WAF) – e.g. CloudFare ($20/m)

Many applications need to be alerted when new documents are inserted or modified in the database. These include real-time apps, or features such as notifications, messaging, or chat. It is possible to build this kind of application with MongoDB, using features that are native to the database: Capped Collections and Tailable Cursors! Details and code samples that illustrate this technique are provided in How to Create a Pub/Sub application with MongoDB. There’s also a screencast that shows the application built in the blog post:

Leading today, VentureBeat reports on the quiet launch of Google’s Cloud Source Repositories. This seems like something we should have heard more about, but I don’t remember seeing anything about it. Amazon AWS announces the availability of all things Alexa, the Skills Kit, the Voice Service and a Fund. Last but not least, we have AJ Kohn, from Blind Five Year Old, talking about click-through rate being a ranking signal on Google’s search results. I don’t talk about SEO much, but reading AJ’s work is always fascinating. As always, enjoy today’s items, and please participate in the discussions on these sites. Top Stories Google has quietly launched a GitHub competitor, Cloud Source Repositories | VentureBeat Alexa Skills Kit, Alexa Voice Service, Alexa Fund | AWS Official Blog Startups, Career and Process Why offices are where work goes to die | Swizec Teller Unleashing the power of small teams | Andreas Papathanasis What Is A Tester? | Developsense Blog What happens when you stop relying on resumes | Aline Lerner Design and Development Swift 2: SIMD | Russ Bishop Why is Git better than Mercurial? | Javalobby Create a Maven archetype | Javalobby pip -t: A simple and transparent alternative to virtualenv | Zoomer Analytics Killing Off Wasabi – Part 1 | Fog Creek Blog WebAssembly- Explained | Modus Create Generating JSON Schema from XSD with JAXB and Jackson | Inspired by Actual Events AI, Machine Learning, Research and Advanced Algorithms Applying Machine Learning to Text Mining with Amazon S3 and RapidMiner | Amazon AWS Big Data, Visualization, SQL and NoSQL Is Click Through Rate A Ranking Signal? | Blind Five Year Old Cache-friendly binary search | Bannalia Discovering the Computer Science Behind Postgres Indexes | Java Code Geeks How an open-source competitive benchmark helped to improve databases | ArangoDB Security, Encryption and Cryptography Cracking JXcore… Again | Mark Haase Link Collections The Daily Six Pack: June 25, 2015 | Dirk Strauss Double Shot #1517 | A Fresh Cup Dew Drop – June 25, 2015 (#2042) | Morning Dew The Daily Six Pack: June 26, 2015 | Dirk Strauss

In this article, we go over several commands developers and data scientists can use to create data frames using existing data frames.

When a team works with a lot of maven projects it becomes quickly painful to do some basic tasks like: manage versionning and connections between the different projects. releasing and opening versions, especially when the maven-release plugin needs to be run on many projects and when versionning is not standard. managing external dependencies also can become complex, and ensuring that a single version of a dependency is used accross different projects is sometimes not a trivial question to ask. in a one word, applying transformations on a dependency graph is difficult. mind-mapping the dependency graph is difficult when the number of projects grows. That can increase the amount of time needed by new people to understand a project graph, and that also makes maintaining and changing things difficult. checking consistency and optimizing the dependency graph is not an easy task neither. having an always up-to-date build of snapshots and release is not easy when projects are distributed everywhere. Pom Explorer So there is the Pom Explorer tool which tries to address those problems by providing those functionalities : release a graph : release a pom or all poms and its/theirs dependencies and updates all dependent poms accross multiple repositories and projects, change a gav : updates a project’s gav and make all the project which depends on it follow this change. manages properties, dependency management, and so on. Pom-explorer knows what pom.xml to update and where to update. If a dependency specifies ${foobar.version}, pom-explorer will go to update the foobar.version property. query the dependency graph to retrieve pertinent information about your projects, statistics and check functions are also available, display 3d interactive graph, export graphml files, find not used dependencies and other similar problems, list java classes provided by artifacts, list java classes referenced by artifacts, runs a light and efficient web server so local and shared usage is possible. The tool will also support automatically building projects in order to always have such or such project always up to date. Use cases In this article, I will show some common use cases possible with this tool. Installation First one needs to install and run the software. Put yourself in a temporary directory and type those commands : git clone https://github.com/ltearno/pom-explorer.git cd pom-explorer java -jar target/pom-explorer.jar The program should welcome you and ask you to go to this address : http://localhost:90 This is the console to the application. You can type commands in the prompt, they will be sent to the server and it will answer. You can use up and down arrows to recall past commands. Let’s start by typing ? to get the available commands : Analyze of repositories OK. First we will analyse a directory where there are many maven projects, then we will work a bit to optimize those projects. You will have to adapt the exercise to your computer. Let’s analyse my git repositories directory : analyze directory c:\documents\repos This will analyze my projects and construct an in-memory graph of the dependency graph : Now, the program knows about everything on my projects, let’s start asking questions ! List of GAVs… Let’s get the list of all existing GAVs (groupId, artifactId, version) in the graph. There will be my projects and all the GAVs on which they depend. Type this command : gav li Note that you can type only the first letters of a command, as long as there is no ambiguity. Here li stands for list. Find dependencies on an obsolete artifact As I look through the list of GAVs, I remark that there are still an old snapshot version of the hexa.binding artifact hanging around. The latest released version is 1.3 and the working version is 1.4-SNAPSHOT so the version 1.3-SNAPSHOTshould not be used anymore. Which is the project still depending on this very deprecated this version ? Let’s ask the question : depends on fr.lteconsulting:hexa.binding:1.3-SNAPSHOT Here it is ! the project rigpa.org:regsys-clients:1.0.0-SNAPSHOT is still using an old snapshot. Let’s arrange that. Pom Explorer is able to change the pom properties and dependencies by itself. Updating this wrong dependency What we want is to change fr.lteconsulting:hexa.binding:1.3-SNAPSHOT tofr.lteconsulting:hexa.binding:1.3 so that the project uses the latest release available. We could desire to change for fr.lteconsulting:hexa.binding:1.4-SNAPSHOT which would be possible with the same command as we’ll see. For that we will use the change gav command : cha ga fr.lteconsulting:hexa.binding:1.3-SNAPSHOT fr.lteconsulting:hexa.binding:1.3 Here is what Pom Explorer answers : So first Pom Explorer finds what needs to be changed in the graph. This might be the project itself and all projects which depend on it. After that the program begins a loop in which all changes are checked and appropriately transformed when needed. For instance changing a dependency version can become changing a property value. Changes are first resolved as described before and they are then transformed in a change list to apply to be applied to pom.xml files. In the ouput, there is first a little warning saying thefr.lteconsulting:hexa.binding:1.3-SNAPSHOT project was not found. That’s normal because the project in now in version 1.4-SNAPSHOT. So there is no need to modify it. Then in the change list section, the changes that are to be applied to pom.xml files are listed. The first one says ‘project not found’ and that’s ok as seen before. The second one says to modify the C:\documents\repos\regsys-clients\pom.xmland change the dependency ([DEPENDENCY])fr.lteconsulting:hexa.binding:1.3-SNAPSHOT tofr.lteconsulting:hexa.binding:1.3. The “causes” message is useful when a change is caused by other changes (as said before a dependency change can become one or several property changes). If we had properties involved, Pom Explorer would have found them and included them in the change set. Now that we reviewed the proposed changes and agreed with them, let’s apply them by using the same command with the -apply flag : cha ga fr.lteconsulting:hexa.binding:1.3-SNAPSHOT fr.lteconsulting:hexa.binding:1.3 -apply We see that at the end of the same process, the program updated the dependency in the right pom.xml file. Let’s have a look at the file it self : fr.lteconsulting hexa.binding 1.3 compile OK, the file is correct now… Oh well no ! I just find other dependencies in SNAPSHOT versions ! Finding more duplicate and obsolete dependencies Let’s accept it, our projects are not up to date. Well let’s see how many of those artifacts there are with multiple versions used. For that i type the checkcommand : Ok there is some work to do ! Opening a version Now let’s look at another use case. Say that the hexa.binding project is in version 1.3 and i want to open the version 1.4-SNAPSHOT. I also want all the projects which depend on version 1.3to move to 1.4-SNAPSHOT. On the way, I want all modified projects still in a release version to be SNAPSHOT-ized too. And i want this to happen recursively as new projects are opened. With Pom Explorer, that’s only one command : change gav fr.lteconsulting:hexa.binding:1.3 fr.lteconsulting:hexa.binding:1.4-SNAPSHOT As you can see, warnings are generated when projects are reopened : Those are normal warnings, they are just here so that you know what happens. Then, there is a big list of changes to be made, because the hexa.bindingartifact is used in many central projects that were in a release state. Glad that we didn’t do that by hand ! Even with the maven-update-version plugin, there would have been a lot of repositories to go to open and update. Let’s apply the changes with the -apply flag : cha ga fr.lteconsulting:hexa.binding:1.3 fr.lteconsulting:hexa.binding:1.4-SNAPSHOT -apply All the changes have been made, about 30 of them. In one go ! Refresh the page so that a new session is created from the changed files. We can see that many of the projects have been reopened : You now have to commit all the repositories with this update. Pom explorer does not do that yet, but maybe in the future ! Releasing many poms Imagine the sprint is almost finished now and it’s now time to release the projects. Type the gav li fr.lteconsulting again to get the GAVs list (fr.lteconsulting is my projects package name, so I filter GAVs with that), choose one and let’s release it : fr.lteconsulting:hexa.binding.samples:1.4-SNAPSHOT The thing in the release is to have all direct and transitive dependencies released too. That’s what Pom Explorer checks. It then generates a change list to materialize your requirements. Other use cases Listing provided and referenced classes You can ask which Java classes are provided and referenced by GAVs. That’s sometimes a useful information to have. Try those commands : classes providedBy fr.lteconsulting:hexa.css:1.3 classes referencedBy fr.lteconsulting:hexa.css:1.3 Optimizing your project’s dependencies Sometimes, you ask yourself “do I still need this and that dependency ?” but you are not very sure, and since you lack time to investigate, eventually the dependency stays in your project for a long time, causing of course maintenance issues sometime. Let’s have Pom Explorer help us in the quest for the obsolete dependency. garbage dep fr.lteconsulting:carousel:1.0-SNAPSHOT This will give you something like that : You can refer to the project documentation to find how to use those informations. But sure that it can help you give away those useless dependencies ! Other goodies : graphs ! Pom Explorer can do two other things to help you visualize your dependency graph : export GraphML files so you can use them in another graph software (like yEd for instance). display an interactive 3d graph Exporting GraphML files GraphML is an open format to describe graphs. With the graph exportcommand, you can get graphml files of your working session. The program will create two files and display the links to them. Those two files are corresponding to two graphs : the dependency graph as usual, and the dependency graph between the git repositories containing your projects. Sometime one git repository can contain multiple projects and a view of the dependencies at the repository level is useful in those cases. This is the kind of picture you can get easily from editors like yEd : Interactive 3D graph Thanks to the WebGL standard which allows direct access to the 3D hardware on the running machine and thanks to libraries like three.js and ngraph.pixel, it is possible to display an interacive 3d graph. More over it is possible to customize the appearance of the graph to give account of different perspectives. Type the graph command and click on the link. This will open another tab containing the living 3d graph of your projects. When focus is given to the 3d viewport, the W, A, S, F and arrow keys allow to move in the 3d space. On the right, there is a text area where you can edit some javascript callback to customize the graph appearance. You can also stop the moving of the particle with the checkbox at the bottom right of the screen. It is not necessarilly useful, but sometimes it is relaxing to admire your work in the form of a living and moving graph ! Conclusion There are many other functions in Pom Explorer, but they are for you to discover now. This tool finds easily its place in the daily workflow because of the functions it provides. The fact that one can run it locally or on a shared server allows to use it as you wish. It is still in early development phase so many more functionalities could come up. On this subject, don’t hesitate submitting a little pull request on the GitHub repository… Pom Explorer is made with love by LTE Consulting

Over the past year, we have had many articles suggesting how Window 10 would be free for anyone who signed up to test it. Well, it turns out that some terms and conditions govern who gets a new Windows 10 license for free. In fact, what you do with your current subscription to Windows Insider Program dictates if you will have to pony up for a new license. Windows Insider Program will continue after the release of Windows 10 on July 29, 2015, essentially meaning beta testers would have two ways to go about their subscription to Windows Insider Program: 1. You continue to remain a part of Windows Insider Program If you decide to remain a part of Windows Insider program, there is nothing extra you have to do. You will continue to receive steady series of pre-release updates. And in return, you will continue to provide end-user feedback to Microsoft on their builds and act as a beta tester. The other important point to note is that you cannot continue using the current Windows 10 preview, while opting out of future updates, as your license would expire. As evident, Microsoft is pushing hard for continued success of Windows Insider Program and as a result wants testers to stay signed up for future pre-releases. You would thereby be required to ensure your test systems are constantly and that you never hit the expiration on the license, essentially meaning you are on an unstable pre-production release with plenty of bugs. 2. You decide to opt out of Windows Insider Program In case you decide opting out of Windows Insider Program, you get same treatment as other Windows customers. The process looks something as shown below. You would need to reserve a spot for free upgrade to Windows 10. However, the catch here is that you should have previously installed Windows 10 Preview on a device running a Windows 7 or 8.1. So if you are someone who is running Windows XP or Vista, you would have to pay in order to get Windows 10 RTM (the final build). Biggest Beta Version of Windows 10 To sum it up, if you want a stable Windows 10 build, you should be running Windows 7 or 8.1. Also, to get a free copy of Windows 10, you can continue being a part of Windows Insider program and thereby be on latest and buggiest beta version of Windows 10. As Milton Friedman rightly said “There’s no such thing as a free lunch in this world”.

Caching on the client side is one of the foundations of World Wide Web. Server should inform client about validity of resources and client should cache them as eagerly as possible. Without caching the web as we see it would be insanely slow. Just hit Ctrl + F5on any website and compare it with ordinary F5 - the latter is much faster as it uses already cached resources. Caching is also important for downloading. If we already fetched several megabytes of data and they haven't changed, pushing them through network is quite wasteful. Use ETag and If-None-Match headers HTTP ETag header can be used to avoid repeatable downloads of resources client already has. Along with first response server returns an ETag header, which is typically a hash value of the contents of a file. Client can keep ETag and send it (in If-None-Matchrequest header) when requesting the same resource later. If it wasn't changed in the meantime, server can simply return 304 Not Modified response. Let's start with an integration test for ETag support: def 'should send file if ETag not present'() { expect: mockMvc .perform( get('/download/' + FileExamples.TXT_FILE_UUID)) .andExpect( status().isOk()) } def 'should send file if ETag present but not matching'() { expect: mockMvc .perform( get('/download/' + FileExamples.TXT_FILE_UUID) .header(IF_NONE_MATCH, '"WHATEVER"')) .andExpect( status().isOk()) } def 'should not send file if ETag matches content'() { given: String etag = FileExamples.TXT_FILE.getEtag() expect: mockMvc .perform( get('/download/' + FileExamples.TXT_FILE_UUID) .header(IF_NONE_MATCH, etag)) .andExpect( status().isNotModified()) .andExpect( header().string(ETAG, etag)) } Interestingly there is a built-in ShallowEtagHeaderFilter in Spring framework. Installing it makes all the tests pass, including the last one: @WebAppConfiguration @ContextConfiguration(classes = [MainApplication]) @ActiveProfiles("test") class DownloadControllerSpec extends Specification { private MockMvc mockMvc @Autowired public void setWebApplicationContext(WebApplicationContext wac) { mockMvc = MockMvcBuilders.webAppContextSetup(wac) .addFilter(new Sha512ShallowEtagHeaderFilter(), "/download/*") .build() } //tests... } I actually plug in my own Sha512ShallowEtagHeaderFilter that uses SHA-512instead of default MD5. Also the default implementation for some reason prepends 0 in front of hash: public class ShallowEtagHeaderFilter { protected String generateETagHeaderValue(byte[] bytes) { StringBuilder builder = new StringBuilder("\"0"); DigestUtils.appendMd5DigestAsHex(bytes, builder); builder.append('"'); return builder.toString(); } //... } vs.: public class Sha512ShallowEtagHeaderFilter extends ShallowEtagHeaderFilter { @Override protected String generateETagHeaderValue(byte[] bytes) { final HashCode hash = Hashing.sha512().hashBytes(bytes); return "\"" + hash + "\""; } } Unfortunately we cannot use built-in filters in our case as they must first fully read response body in order to compute ETag. This basically turns off body streaming introduced in previous article - whole response is stored in memory. We must implement ETagfunctionality ourselves. Technically If-None-Match can include multiple ETag values. However neither Google Chrome nor ShallowEtagHeaderFilter support it, so we will skip that as well. In order to control response headers we now returnResponseEntity: @RequestMapping(method = GET, value = "/{uuid}") public ResponseEntity download( @PathVariable UUID uuid, @RequestHeader(IF_NONE_MATCH) Optional requestEtagOpt) { return storage .findFile(uuid) .map(pointer -> prepareResponse(pointer, requestEtagOpt)) .orElseGet(() -> new ResponseEntity<>(NOT_FOUND)); } private ResponseEntity prepareResponse(FilePointer filePointer, Optional requestEtagOpt) { return requestEtagOpt .filter(filePointer::matchesEtag) .map(this::notModified) .orElseGet(() -> serveDownload(filePointer)); } private ResponseEntity notModified(String etag) { log.trace("Cached on client side {}, returning 304", etag); return ResponseEntity .status(NOT_MODIFIED) .eTag(etag) .body(null); } private ResponseEntity serveDownload(FilePointer filePointer) { log.debug("Serving '{}'", filePointer); final InputStream inputStream = filePointer.open(); final InputStreamResource resource = new InputStreamResource(inputStream); return ResponseEntity .status(OK) .eTag(filePointer.getEtag()) .body(resource); } The process is controlled by optional requestEtagOpt. If it's present and matches whatever was sent by the client, we return 304. Otherwise 200 OK as usual. New methods in FilePointer introduced in this example look as follows: import com.google.common.hash.HashCode; import com.google.common.hash.Hashing; import com.google.common.io.Files; public class FileSystemPointer implements FilePointer { private final File target; private final HashCode tag; public FileSystemPointer(File target) { try { this.target = target; this.tag = Files.hash(target, Hashing.sha512()); } catch (IOException e) { throw new IllegalArgumentException(e); } } @Override public InputStream open() { try { return new BufferedInputStream(new FileInputStream(target)); } catch (FileNotFoundException e) { throw new IllegalArgumentException(e); } } @Override public String getEtag() { return "\"" + tag + "\""; } @Override public boolean matchesEtag(String requestEtag) { return getEtag().equals(requestEtag); } } Here you see FileSystemPointer implementation that reads files straight from file system. The crucial part is to cache tag instead of recalculating it on every request. The implementation above behaves as expected, for example web browsers won't download the resource again. 3. Use Last-Modified header Similar to ETag and If-None-Match headers there are Last-Modified and If-Modified-Since. I guess they are pretty self-explanatory: first server returns Last-Modified response header indicating when a given resource was last modified (duh!). Client caches this timestamp and passes it along with subsequent request to the same resource in If-Modified-Since request header. If the resource wasn't changed in the meantime, server will respond with 304, saving bandwidth. This is a fallback mechanism and it's a good practice to implement both ETags and Last-Modified. Let's start with integration tests: def 'should not return file if wasn\'t modified recently'() { given: Instant lastModified = FileExamples.TXT_FILE.getLastModified() String dateHeader = toDateHeader(lastModified) expect: mockMvc .perform( get('/download/' + FileExamples.TXT_FILE_UUID) .header(IF_MODIFIED_SINCE, dateHeader)) .andExpect( status().isNotModified()) } def 'should not return file if server has older version than the client'() { given: Instant lastModifiedLaterThanServer = FileExamples.TXT_FILE.getLastModified().plusSeconds(60) String dateHeader = toDateHeader(lastModifiedLaterThanServer) expect: mockMvc .perform( get('/download/' + FileExamples.TXT_FILE_UUID) .header(IF_MODIFIED_SINCE, dateHeader)) .andExpect( status().isNotModified()) } def 'should return file if was modified after last retrieval'() { given: Instant lastModifiedRecently = FileExamples.TXT_FILE.getLastModified().minusSeconds(60) String dateHeader = toDateHeader(lastModifiedRecently) expect: mockMvc .perform( get('/download/' + FileExamples.TXT_FILE_UUID) .header(IF_MODIFIED_SINCE, dateHeader)) .andExpect( status().isOk()) } private static String toDateHeader(Instant lastModified) { ZonedDateTime dateTime = ZonedDateTime.ofInstant(lastModified, ZoneOffset.UTC) DateTimeFormatter.RFC_1123_DATE_TIME.format(dateTime) } And the implementation: @RequestMapping(method = GET, value = "/{uuid}") public ResponseEntity download( @PathVariable UUID uuid, @RequestHeader(IF_NONE_MATCH) Optional requestEtagOpt, @RequestHeader(IF_MODIFIED_SINCE) Optional ifModifiedSinceOpt ) { return storage .findFile(uuid) .map(pointer -> prepareResponse( pointer, requestEtagOpt, ifModifiedSinceOpt.map(Date::toInstant))) .orElseGet(() -> new ResponseEntity<>(NOT_FOUND)); } private ResponseEntity prepareResponse(FilePointer filePointer, Optional requestEtagOpt, Optional ifModifiedSinceOpt) { if (requestEtagOpt.isPresent()) { final String requestEtag = requestEtagOpt.get(); if (filePointer.matchesEtag(requestEtag)) { return notModified(filePointer); } } if (ifModifiedSinceOpt.isPresent()) { final Instant isModifiedSince = ifModifiedSinceOpt.get(); if (filePointer.modifiedAfter(isModifiedSince)) { return notModified(filePointer); } } return serveDownload(filePointer); } private ResponseEntity serveDownload(FilePointer filePointer) { log.debug("Serving '{}'", filePointer); final InputStream inputStream = filePointer.open(); final InputStreamResource resource = new InputStreamResource(inputStream); return response(filePointer, OK, resource); } private ResponseEntity notModified(FilePointer filePointer) { log.trace("Cached on client side {}, returning 304", filePointer); return response(filePointer, NOT_MODIFIED, null); } private ResponseEntity response(FilePointer filePointer, HttpStatus status, Resource body) { return ResponseEntity .status(status) .eTag(filePointer.getEtag()) .lastModified(filePointer.getLastModified().toEpochMilli()).body(body); } Sadly using Optional idiomatically no longer looks good so I stick to isPresent(). We check both If-Modified-Since and If-None-Match. If neither match, we serve file as usual. Just to give you a taste of how these headers work, let's execute few end-to-end tests. First request: > GET /download/4a8883b6-ead6-4b9e-8979-85f9846cab4b HTTP/1.1 > ... > < HTTP/1.1 200 OK < ETag: "8b97c678a7f1d2e0af...921228d8e" < Last-Modified: Sun, 17 May 2015 15:45:26 GMT < ... Subsequent request with ETag (shortened): > GET /download/4a8883b6-ead6-4b9e-8979-85f9846cab4b HTTP/1.1 > If-None-Match: "8b97c678a7f1d2e0af...921228d8e" > ... > < HTTP/1.1 304 Not Modified < ETag: "8b97c678a7f1d2e0af...921228d8e" < Last-Modified: Sun, 17 May 2015 15:45:26 GMT < ... And in case our client supports Last-Modified only: > GET /download/4a8883b6-ead6-4b9e-8979-85f9846cab4b HTTP/1.1 > If-Modified-Since: Tue, 19 May 2015 06:59:55 GMT > ... > < HTTP/1.1 304 Not Modified < ETag: "8b97c678a7f1d2e0af9cda473b36c21f1b68e35b93fec2eb5c38d182c7e8f43a069885ec56e127c2588f9495011fd8ce032825b6d3136df7adbaa1f921228d8e" < Last-Modified: Sun, 17 May 2015 15:45:26 GMT There are many built-in tools such as filter that can handle caching for you. However if you need to be sure your files are streamed rather then pre-buffered on the server side, extra care needs to be taken. Writing a download server Part I: Always stream, never keep fully in memory Part II: headers: Last-Modified, ETag and If-None-Match Part III: headers: Content-length and Range Part IV: Implement HEAD operation (efficiently) Part V: Throttle download speed Part VI: Describe what you send (Content-type, et.al.) The sample application developed throughout these articles is available on GitHub.

CryTek just released its CryEngine 3.8.1, an update packed with features. The CryEngine website dubs this the heftiest upgrade since debuting their Engine-as-a-Service in May. Among the many new features is virtual reality (VR) support, OpenGL compatibility, and Linux support. One of the biggest trends in gaming is VR, and CryEngine 3.8.1 adds VR support. Initially, it’s limited to the Oculus Rift, but chances are as more headsets emerge and see adoption among both developers and gamers, compatibility will expand to support these as well. Epic Games’ Unreal Engine also added VR support, along with Unity3D’s most recent release. There’s a neat VR demo on the CryTek website, which developers will surely want to check out. Another significant change is the addition of Linux support. While Wine and Playonlinux have both helped many games to run on a variety of Linux-based operating systems (OSes). However, native Linux support means easier use for developers. As more games add Linux compatibility, spearheaded by Steam’s SteamOS, the CryEngine itself can now be run on Linux. This latest update means that the CryEngine will join Unity3D, the Unreal Engine, and Source as a powerful game development engine with Linux and VR support. Virtual reality is seeing widespread adoption among the developer community, and Linux compatibility in gaming is a huge trend. While CryEngine 3.8.1’s ability to run on Linux won’t necessarily mean games developed will be compatible on the popular open source OS, it certainly makes it easier to ensure Linux support. There’s also OpenGL support, which will also aid cross-platform development. The CryEngine has been used to create many games known for their gorgeous eye-candy. Notably, the CryTek’s aptly names Crysis series is built with the CryEngine, as is Ryse: Son of Rome, State of Decay, and Kingdom Come: Deliverance.

It’s been a year since I started the quest for a highly-effective Data Knowledge Stack and the Hibernate Master Class contains over fifty articles already. Now that I covered many aspects of database transactions, JDBC and Java Persistence, it’s time to assemble all the pieces together into the High-Performance Java Persistence book. An Agile publishing experience Writing a book is a very time-consuming and stressful process and the last thing I needed was a very tight schedule. After reading Antonio Goncalves’s story, I chose the self-publishing way. In the end, I settled for Leanpub because it allows me to publish the book incrementally. This leads to a better engagement with readers, allowing me adapt the book content on the way. The content At its core, the book is about getting the most out of your persistence layer and that can only happen when your application resonates with the database system. Because concurrency is inherent to database processing, transactions play a very important role in this regard. The first part will be about some basic performance-related database concepts such as: locking, batching, connection pooling. In the second part, I will explain how an ORM can actually improve DML performance. This part will include the Hibernate Master Class findings. The third part is about advance querying techniques with jOOQ. If you enjoy reading this article, you might want to subscribe to my newsletter and get a discount for my book as well. Get involved The Agile methodologies are not just for software development. Writing a book in a Lean style can shorten the feed-back period and readers can get involved on the way. If you have any specific request or you are interested in this project, you can join my newsletter and follow my progress. Buy it! The book is 100% done, and you can check out the full Table of Content onLeanpub. If you enjoyed this article, I bet you are going to love my book as well. The ebook The PDF, ePUB and Kindle (MOBI) versions can be bought on Leanpub. The print version The print version is available on Amazon, Amazon.co.uk, Amazon.de or Amazon.fr. Presentations If you are not convinced, then check out the following two presentations: High-Performance JDBC from Voxxed Days Bucharest High-Performance Hibernate from JavaZone

A blue-green deployment is a way to have incremental updates to your production stack without downtime and without any complexity for properly handling rolling updates (including the rollback functionality) I don’t need to repeat this wonderful explanation or Martin Fowler’s original piece. But I’ll extend on them. A blue-green deployment is one where there is an “active” and a “spare” set of servers. The active running the current version, and the spare being ready to run any newly deployed version. The “active” and “spare” is slightly different than “blue” and “green”, because one set is always “blue” and one is always “green”, while the “active” and “spare” labels change. On AWS, for example, you can script the deployment by having two child stacks of your main stacks – active and spare (indicated by a stack label), each having one (or more) auto-scaling group for your application layer, and a script that does the following (applicable to non-AWS as well): push build to an accessible location (e.g. s3) set the spare auto-scaling group size to the desired value (the spare stays at 0 when not used) make it fetch the pushed build on startup wait for it to start run sanity tests switch DNS to point to an ELB in front of the spare ASG switch the labels to make the spare one active and vice versa set the previously active ASG size to 0 The application layer is stateless, so it’s easy to do hot-replaces like that. But (as Fowler indicated) the database is the most tricky component. If you have 2 databases, where the spare one is a slave replica of the active one (and that changes every time you switch), the setup becomes more complicated. And you’ll still have to do schema changes. So using a single database, if possible, is the easier approach, regardless of whether you have a “regular” database or a schemaless one. In fact, it boils down to having your application modify the database on startup, in a way that works with both versions. This includes schema changes – table (or the relevant term in the schemaless db) creation, field addition/removal and inserting new data (e.g. enumerations). And it can go wrong in many ways, depending on the data and datatypes. Some nulls, some datatype change that makes a few values unparseable, etc. Of course, it’s harder to do it with a regular SQL database. As suggested in the post I linked earlier, you can use stored procedures (which I don’t like), or you can use a database migration tool. For a schemaless database you must do stuff manually, but but fewer actions are normally needed – you don’t have to alter tables or explicitly create new ones, as everything is handled automatically. And the most important thing is to not break the running version. But how to make sure everything works? test on staging – preferably with a replica of the production database (automatically) run your behaviour/acceptance/sanity test suites against the not-yet-active new deployment before switching the DNS to point to it. Stop the process if they fail. Only after these checks pass, switch the DNS and point your domain to the previously spare group, thus promoting it to “active”. Switching can be done manually, or automatically with the deployment script. The “switch” can be other than a DNS one (as you need a low TTL for that). It can be a load-balancer or a subnet configuration, for example – the best option depends on your setup. And while it is good to automate everything, having a few manual steps isn’t necessarily a bad thing. Overall, I’d recommend the blue-green deployment approach in order to achieve zero downtime upgrades. But always make sure your database is properly upgraded, so that it works with both the old and the new version.

Are you really sure what makes a programmer productive? Is it VIM instead of Emacs, the latest Haskell web framework or your favourite NoSQL database? Sorry, but if you focus on tools, frameworks or even processes, you got it backwards! Real programmer productivity starts at the very beginning: Proper requirements. Why you as a developer must care - not only the business staff ! Sure, the founder/product owner/team lead/name of the month must care to build something that a customer ultimately pays the company money for. But what about it from a developer’s view? Ever been in the situation where someone shouts over the table: „Just start building XYZ right away, if any questions pop up, we will deal with them during development. We got a head start anyway“ ? We call that: Start first, finish never. Nothing beats building something, where half of what you are building is actually still unclear. How do you know you are done DONE? Not surprisingly, not knowing 100% „what“ to do, leaks a lot into „when are you finished?“ „Oh, we just forgot this…and that!..And actually it should also be doing this! That point of the feature…dunno really?“ And how do you go about testing unclear requirements? This has nothing to do with your favourite BDD tool of the month, or your freaking out if someone does not always test first. (Even though we think there are some, let’s call them preferences, that make building software easier.). If the input is unclear, tests are unclear and the output is even more unclear. You are always super-motivated, right? But the worst part is, that frequent unclear requirements are a sign. A sign that your business people are unsure of what your customers really want/need/pay for. Unfortunately this also means, that a lot of the stuff you build is for the trash can. Nothing keeps programmer motivation and ultimately productivity higher than repeatedly baking bread and throwing it in the trash while it is still hot. What exactly is a proper requirement ? Now what exactly is a proper requirement? Is it a sentence that is written on an index card and starts with „as a user I want to be able to use my Asian CCB credit card?“. Nope. We actually came up with a fancy sounding process for generating and checking proper requirements. We will go into detail on the individual points in our next blog articles, but here’s the abstract spoiler for what a proper requirement is. (Oh, and we'll spice it up with real-life examples from the requirements engineering for the new BMW (car manufacturer) website a while ago.) A proper requirement has (1) been worked out with business people AND programmers, with two-way street communication. It has (2) been repeatedly deconstructed. deconstructed. deconstructed. And it has (3) been defended, which includes what we call „angling“ and „skinning“. STOP it already ! I'm a programmer, requirements are not my job! Yes, in bigger organisations you most often have dedicated business analysts, whose sole job is to iron out detailed requirement specifications before they get passed to „implementation teams“. And in dreamland this all works flawlessly, and you just sit down and start coding, but in reality not so much. Anyway, the smaller an organisation gets, the more a programmer becomes a hybrid. The founder might shout across the table: And you as a „programmer“ have to not only sort out the implementation, but also what the hell this is all about. In any case, you should be a professional ! As much fun as it might be to read through he upgrade path of AngularJS 2.0 instead of accustoming yourself with your customers problem domain and requirements : At the end of the day, as sad as it might seem, your technical skills, voice of frameworks or algorithms are only a part of your day-to-day job. And the basis for all development work is : proper requirements. Next Step: Share your experiences in the comment section or on our blog, subscribe to our newsletter and stay tuned for the follow-up articles!

[This article was written by Matt Kiernan] We are excited to announce that Logentries’ new SQL-like query language, LEQL, is now available for more advanced analytics and easy extraction of valuable insights from your log data. A SQL-Like Query Language If you’ve ever used SQL, LEQL should feel familiar. In fact, Logentries already supports a number of SQL-like search functions, including: SUM: Sums a set of values COUNT: Counts the number of times a value occurs GROUPBY: Groups values by a unique key UNIQUE: Enables the count of only unique values With the rollout of LEQL, we’ll be introducing four new query functions: MIN: Calculate the minimum value of a specified key MAX: Calculate the maximum value of a specified key SORT: Display results sorted either ascending or descending TIMESLICE: Specifies how to group by time (e.g. by specific number of minutes, hours or days) A Consistent Yet Expressive Syntax We believe a reliable query language depends on a consistently enforced syntax. For this reason, we’ll be enforcing how queries are structured. Here’s an example of how an old query would change with LEQL: Old pages>0 | GroupBY(dbName) | SUM(pages) New where(pages>0) groupby (dbName) calculate(SUM:pages) *In this example, pages & dbName are Key names in log events Notice how the search logic gets wrapped in a where() clause, used for refining your search to return only results that match your search criteria (i.e. where events include the text or Key “pages”.) groupBy() is an optional clause that enables you to organize your search results into groups by specifying a Key from a Key-Value Pair (i.e. key: value). Calculations made within your query get utilized in the calculate()clause. When building your query, you no longer need to separate sections with pipes “|”. Though we believe in the value of a consistent query syntax, we also believe in the importance of giving users an expressive language that is easy to use and delivers expected results. We’re taking the following steps to make LEQL easy to use: Outdated saved queries will automatically be converted into LEQL – no effort required where clauses will automatically be added to any new query you write LEQL terms will not be case sensitive An updated search bar will provide a query builder and validator An updated search bar & query builder An Updated Search Bar & Query Builder As we rollout LEQL, we’ll be introducing a new search bar, allowing users to switch between a simple & advanced modes based on their preference. Simple mode “Simple mode” provides an easy way to build queries by providing a list of the available functions. Type-Assist will show a list of keys to associate with each functions, or new keys can be typed manually. Advanced mode “Advanced mode” will allow users to type their queries manually. Type-Assist will autocomplete key names while the new search bar will automatically validate query syntax. July 1st Rollout The LEQL rollout will take place in phases, beginning July 1st and will continue over the next few weeks to update all plans. If you’d like early beta access to LEQL, or have any questions, feel free to reach us at [email protected].

[This article was written by Wyatt Lindsay] New Relic’s Community Forum is designed to be a place for our users to share their experiences, questions, problems, and fixes. The collective expertise and creativity of the New Relic community has generated some outstanding solutions to everyday issues, and we want to call out some of them in the area of .NET agent deployment excellence. Basic installation Installing New Relic’s .NET agent is designed to be simple: run the installer on the target host and choose the features you want to include. For Microsoft Azuredeployments, install one of our NuGet packages. Installation requires a reset of IIS for the software to load into your application. To upgrade, we recommend first stopping IIS, installing the newer agent version, and then starting up IIS again. It’s also possible to perform a “silent” (manual) install using msiexec.exe, for example: msiexec.exe /i C:\NewRelicAgent.msi /qb NR_LICENSE_KEY= INSTALLLEVEL=1 See the documentation for complete manual installation options. Leveraging PowerShell Scripting the silent installation provides convenience and flexibility. Here’s an example script provided by community member Jon Carl in this post: $msiName = $licenseKey = $arguments = "/i $msiName /L*v install.log /qn NR_LICENSE_KEY=$licenseKey" if ($msiName -ne $null) { $exitCode = (Start-Process -FilePath "msiexec" -ArgumentList $arguments -Wait -PassThru).ExitCode; if($exitCode -eq 0) { Write-Host "Installation successful!" -ForegroundColor Green } else { Write-Host "Installation unsuccessful. Exitcode: $exitCode" -ForegroundColor Red } } This script works great when run directly on the target machine. Another forum user (Kym McGain) noticed that the installation didn’t complete before the session ended when executing the script remotely. This caused the installer to quit partway through. Kym posted this script that uses a ‘while’ loop to ensure the installer completes. As a bonus, it stops IIS before and restarts it after the software installs. As mentioned above, these steps are usually needed when upgrading. $installNewRelic = { $runProcess = { param($process,$arguments) $res = Start-Process -FilePath $process -ArgumentList $arguments -Wait -PassThru while ($res.HasExited -eq $false) { Write-Host "Waiting for $process..." Start-Sleep -s 1 } $exitCode = $res.ExitCode if($exitCode -eq 0) { Write-Host "$process successful!" -ForegroundColor Green } else { Write-Host "$process unsuccessful. Exitcode: $exitCode" -ForegroundColor Red } } $msiName = $licenseKey = $arguments = "/i $msiName /L*v install.log /qn NR_LICENSE_KEY=$licenseKey" Invoke-Command $runProcess -ArgumentList "IISRESET","/STOP" Invoke-Command $runProcess -ArgumentList "msiexec.exe",$arguments Invoke-Command $runProcess -ArgumentList "IISRESET","/START" } Chef, Puppet, and Chocolatey Deployment options abound for modern Web developers. These solutions often require a known download path and installer name. New Relic offers a consistent filepath and MSI name for the agent in an effort to make automated deployment easier for .NET customers: http://download.newrelic.com/dot_net_agent/release/NewRelicDotNetAgent_x64.msi http://download.newrelic.com/dot_net_agent/release/NewRelicDotNetAgent_x86.msi Several Community members have created packages for these utilities. Chocolatey users are invited to use the following NuGet package created by kireevco: https://chocolatey.org/packages/newrelic-dotnet New Relic community member ePitty built a Puppet module to handle .NET agent deployment: https://github.com/epitty1023/puppet-newrelicappmon Chef users, meanwhile, can check out the following cookbook for .NET and many other platforms New Relic supports: http://community.opscode.com/cookbooks/newrelic New Relic Community member E_Bow wrote a Chef recipe that goes a step further by stopping IIS before the installation and starting it again after completion: #Stop IIS iis_site 'Website' do action [:stop] end # install latest Newrelic agent from web include_recipe 'newrelic::repository' include_recipe node['newrelic']['dotnet-agent']['dotnet_recipe'] license = node['newrelic']['application_monitoring']['license'] windows_package 'Install New Relic .NET Agent' do source node['newrelic']['dotnet-agent']['https_download'] options "/qb NR_LICENSE_KEY=#{license} INSTALLLEVEL=#{node['newrelic']['dotnet-agent']['install_level']}" installer_type :msi action :install end #Start IIS iis_site 'Website' do action [:start] end The author states that they were unable to pull the New Relic license key from the configuration JSON in Chef Overrides, requiring them to modify the config file on each machine and manually enter the key. We invite any Chef experts out there to extend and improve this recipe so that it correctly pulls the license key. We are continually impressed by the smarts and spirit of our New Relic Community Forum members, and jump at the chance to highlight their contributions. Look for more Forum projects in the New Relic blog in the future. Do you have your own approach, tips, or recipes? Please share them in the New Relic Community Forum.

This past week, I headed to London for the Rugged DevOps event, where I had the chance to catch up with a few more industry thought leaders. First, I caught up with Gareth Rushgrove from Puppet Labs, who also runs the DevOps Weekly newsletter. In this episode of the DevOps Leadership Series, Gareth explains why the importance of security tests within DevOps practices is going to be a big topic for years to come. Gareth anticipates that “In five years time we will look back on this and ask ‘why were we not always doing this’?” I then caught up with Helen Beal, Head of DevOps at Ranger4, where she discussed the importance of DevOps and security. While the two ultimately need to work hand and hand, she voiced concerns about DevOps practices sometimes circumventing controls that are essential to a business’s safety. She also said that DevOps supports security in a number of ways: from making things consistent to relying on more automation. Finally, I caught up with Justin Arbuckle, Vice President, EMEA & Chief Enterprise Architect at Chef. He discusses key trends he expects to see in over the next 9-12 months regarding compliance and security. Justin explains that compliance as a core driver of DevOps is something that will start to shape our conversations over the next year, whereas the two have previously been seen as contradictory. He believes that we will see that high velocity organizations are able to improve compliance continuously, and tells us to look out for the changing role of the security officer in 2015. Next up in the series, I head to the United States Capitol, Washington DC, for DevOps Days DC. NOTE: If you have missed any of the other videos from this series, you can find them here. (We’re up to 15 so far).

Occasionally we are asked about JBoss BPM Suite integration with other products and layers in an enterprises architecture. We have published articles talking about how to achieve this with various aspects such as: Microservices integration Data integration Articles are one thing, but seeing is believing, so we have done a few webinars to show you live how to tackle integration: Data integration webinar PEX webinar Along with these articles we have always published demo projects that give you a closer look and chance to get hands on with these integration strategies: JBoss BPM Suite & JBoss Fuse Travel microservices story JBoss BPM Suite & JBoss Data Virtualization integration Imported Signavio Process Editor mortgage workflow. There is another integration story yet to be told about how one can leverage other tooling together with JBoss BPM Suite. This article will introduce one such company,Signavio, that provides a Signavio Process Editor so"...you can start modeling and engaging your organization in improving operational efficiency through the development of optimal models..." The following demo project provides a working example of how you can model an example mortgage process in Signavio Process Editor and then bring it into JBoss BPM Suite where you can add implementation details, integration details and other implementation details to finally execute the mortgage process end-to-end. Demo project As always we bring you not only a story, but a reusable demo project you can easily spin up yourself to explore the details around how a JBoss BPM project would integrate with the model designed in Signavio Process Editor. The project is called the JBoss BPM Suite & Signavio Process Editor Integration Demo. The project installs JBoss BPM Suite 6.1 with an example mortgage project with rules, process, forms and other artifacts. It also includes a copy of an exported Signavio Process Editor mortgage process that we then show how to import. Final mortgage workflow project with implementation details and integration details completed. Ready to run! This gives you the initial starting point after importing the Signavio process and the completely integrated final mortgage project that you can run side-by-side. To setup this project there are just a few simple steps to get going and will be up and running minutes: Installation Download and unzip. Add products to installs directory. Run 'init.sh' or 'init.bat' file. 'init.bat' must be run with Administrative privileges. Start JBoss BPMS Server by running 'standalone.sh' or 'standalone.bat' in the /target/jboss-eap-6.1/bin directory. Login to http://localhost:8080/business-central - login for admin, appraisor, broker, and manager roles (u:erics / p:bpmsuite1!) Mortgage Loan demo pre-installed as project. Using process designer, import the Signavio process that was exported to the file found in: support/MortgageDemoSignavio.bpmn Looking to Automate your business? See screenshots provided in project for how this should look and note that the JBoss BPM Suite process designer included validation that puts messages about tasks not specified, this is correct as at this point you need to start implementing the process tasks. You can examine the imported process and note the various details captured during initial workshops have been put into the process details for each step in the workflow. After implementing these steps you will find the final process ready to run. You can now explore the final project by deploying it and starting a new instance. We hope you enjoy this example project and feel free to browse for more at JBoss Demo Central.

The use of OpenStack in the private cloud is invaluable for DevOps. It provides engineers the ability to innovate quickly and deal with uncertainty. It also maximizes existing infrastructure and provides a programmable, software-defined IaC. Openstack in the private cloud = agile development OpenStack has emerged as the de facto standard for IaaS in the private cloud. It gives engineers a vital self-service capability to provision (and de-provision) environments, allowing them to act autonomously, in the moment. This helps to eliminate the downstream bottleneck caused by waiting for operations staff to find time to do the provisioning. As OpenStack is open source it is vendor agnostic, allowing you to take advantage of competitive pricing rather than suffering from vendor lock-in. A private cloud means lower cost for the same capacity in a public cloud, which is especially useful for enterprises with high data needs. For security reasons, OpenStack is still mainly used in the private cloud by developers and QA, i.e. in a non-production context. However, OpenStack gives an ability to optimize application performance and/or security by having more control compared to public cloud. The software is increasingly backed by the critical mass of leading IT infrastructure vendors such as IBM, CICSO and HP. Gartner assumes that “by 2019, OpenStack enterprise deployments will grow tenfold, up from just hundreds of production deployments today, due to increased maturity and growing ecosystem support.”1 Challenges to consider OpenStack implementation skills are still rare in the market, so experimentation and self-learning is necessary. Although this takes time, it is offset by the fact the software is free and represents a good opportunity to gain internal expertise. This is particularly valid if you class infrastructure as a core competence. The maturity and functionality of OpenStack projects vary widely - while it covers storage, network and compute, the main adoption currently happens around compute (Nova) and block storage (Cinder), with object storage and network (Neutron) lacking significantly behind. However, without leveraging virtualized network services as part of a private cloud, full-stack environment provisioning is not possible, so don’t forget to add necessary network services to your private cloud. Where to begin Integrating OpenStack clouds with existing infrastructure can be a challenge. It is hardly plug and play. At first, it is best to focus on relatively isolated DevOps environments, such as Gartner’s “mode two”2 applications rather than introducing open stack across the board straight away, (Bimodal IT “refers to having two modes of IT, each designed to develop and deliver information – and technology – intensive services in its own way. Mode 1 is traditional, emphasizing scalability, efficiency, safety and accuracy. Mode 2 is nonsequential, emphasizing agility and speed.”3) As with any open source software, new functions and upgrades are frequently released. This means keeping up with changes in functionality and filling gaps with customizations or third-party products. Upgrades are complex and typically require planned downtime. For these reasons, we recommend choosing a hardened distribution and sticking with it. Openstack is the most complete vendor agnostic solution for storage, network and compute services. The ability for developers to instantly spin up environments at any time is invaluable for a fully agile DevOps environment, and is well worth the effort it takes to acclimatize to Openstack. 1 http://www.prnewswire.com/news-releases/suse-openstack-cloud-5-to-simplify-private-cloud-management-300048721.html 2 http://www.gartner.com/it-glossary/bimodal 3 http://www.gartner.com/it-glossary/bimodal

business sphere nowadays has become an extremely competitive arena. dynamics change in a blink. times have become highly unpredictable and hence; businesses today need to be agile while being equipped with reliable, accurate, relevant and actionable business intelligence. every business venture has its own fair share of ebbs and tides. it becomes more of a challenge to prove your capabilities and achieve a strong hold in the market; especially when you have just started taking your first step in. for startups, getting the minutest nuances of how to run a business; right from the day one, forms the most crucial part! to smoothly sail through this enormously competitive space; startups need to perform above and beyond the expectations right from the very beginning. the initial barriers can be easily overcome when your business is armed with smallest details of the market. but how to catch the nerve of market, you will ask? - data extraction or data mining services is the answer! data mining equips you with rich business intelligence that in turn gives a firm control of things and empowers you to make informed business decisions as well as create more targeted, applicable and growth-oriented business strategies. data extraction services gather huge volume of data that is highly varied, precise, and relevant. most importantly - it is very useful for your new startup . a meticulous study of this database allows you to analyze things in great details and arranging this scattered information into meaningful clusters; helps you get the whole picture! which are the different ways for startups to effectively use web data mining? web data mining is a wide array, which can be employed for a variety of purposes to generate various kinds of important data to gain actionable insights. in fact, for a startup, the most critical part is to decide where and how to use this powerful technique to get valuable information which can help in creating a difference for overall future prospects of the company. let’s check out on some of those interesting avenues; where you can apply impactful web data extraction techniques: digging information for social rankings and backlinks for any startup; the most crucial business process is to analyze its competitors. this is one area where web data extraction comes across an instrumental enabler. many startups, in the past, have effectively used data mining to fish out critically useful information related to social rankings of competing companies. social ranking is equally important factor, since any ‘social actions’ on the internet are building blocks of several opinions as well as builds a reputation in this day and age. keeping these things in mind, you can use web data extraction to dig out for social rankings related to content created by your competitors in the cyber space. with thorough analysis; you can get a very clear picture of the entire situation and it helps you to arrive to a concrete conclusions in terms of what your competitors are doing well at, and what sells the best. obtaining contact information building strong networking is the best bet which helps you to get through the volatile market; specifically when you are a newbie in the market. whether it is with prospective or existing customers, industry peers, associates, or competitors; excellent networking is the driving force where there is open and transparent communication, ensures success of your startup. and to have such an effective communication and networking channel, you need a huge, robust list of contact information that is in sync with - your exact requirements. mining data from multiple web sources is by all means a perfect method to achieve this. in a short period of time you can easily collect rich contact information that can be leveraged in a number of ways. you can form a long lasting business relationship or make potential customers know what you offer; this information gives a thrust to your startup and propels it to new levels of recognition. for building brand, promotion and advertisement for startups, the very first wave of promotion is the key that builds a strong brand value in the market and ensures long-term business success. it is during this initial phase that the first and foremost public perception of your company is created, and the essentials of public opinion starts shaping up. for this reason, it is required to be precise with your marketing and promotion these formative years. to achieve this, you need a strong, in-depth understanding of the audience that you need to target. you require to classify your target audience based on factors like age, gender, income, demographics, and preferences. such detailed understanding can be attained only when you have a voluminous social data related to the targeted audience. and there is no better way to achieve this, other than web data extraction. with such a powerful weapon in your arsenal, you can certainly boost up your startup and take it a long way with clever decisions and timely implementations. web data extraction can be the absolute tool that a startup may ever have! its appropriate use should give you tons of required and relevant business intelligence, which should help you to shine in this competitive market.

Written by Yves Trudeau Ever since MySQL replication has existed, people have dreamed of a good solution to automatically split read from write operations, sending the writes to the MySQL master and load balancing the reads over a set of MySQL slaves. While if at first it seems easy to solve, the reality is far more complex. First, the tool needs to make sure it parses and analyses correctly all the forms of SQL MySQL supports in order to sort writes from reads, something that is not as easy as it seems. Second, it needs to take into account if a session is in a transaction or not. While in a transaction, the default transaction isolation level in InnoDB, Repeatable-read, and the MVCC framework insure that you’ll get a consistent view for the duration of the transaction. That means all statements executed inside a transaction must run on the master but, when the transaction commits or rollbacks, the following select statements on the session can be again load balanced to the slaves, if the session is in autocommit mode of course. Then, what do you do with sessions that set variables? Do you restrict those sessions to the master or you replay them to the slave? If you replay the set variable commands, you need to associate the client connection to a set of MySQL backend connections, made of at least a master and a slave. What about temporary objects like with “create temporary table…”? How do you deal when a slave lags behind or what if worse, replication is broken? Those are just a few of the challenges you face when you want to build a tool to perform read/write splitting. Over the last few years, a few products have tried to tackle the read/write split challenge. The MySQL_proxy was the first attempt I am aware of at solving this problem but it ended up with many limitations. ScaleARC does a much better job and is very usable but it stills has some limitations. The latest contender is MaxScale from MariaDB and this post is a road story of my first implementation of MaxScale for a customer. Let me first introduce what is MaxScale exactly. MaxScale is an open source project, developed by MariaDB, that aims to be a modular proxy for MySQL. Most of the functionality in MaxScale is implemented as modules, which includes for example, modules for the MySQL protocol, client side and server side. Other families of available modules are routers, monitors and filters. Routers are used to determine where to send a query, Read/Write splitting is accomplished by the readwritesplit router. The readwritesplit router uses an embedded MySQL server to parse the queries… quite clever and hard to beat in term of query parsing. There are other routers available, the readconnrouter is basically a round-robin load balancer with optional weights, the schemarouter is a way to shard your data by schema and the binlog router is useful to manage a large number of slaves (have a look at Booking.com’s Jean-François Gagné’s talk at PLMCE15 to see how it can be used). Monitors are modules that maintain information about the backend MySQL servers. There are monitors for a replicating setup, for Galera and for NDB cluster. Finally, the filters are modules that can be inserted in the software stack to manipulate the queries and the resultsets. All those modules have well defined APIs and thus, writing a custom module is rather easy, even for a non-developer like me, basic C skills are needed though. All event handling in MaxScale uses epoll and it supports multiple threads. Over the last few months I worked with a customer having a challenging problem. On a PXC cluster, they have more than 30k queries/s and because of their write pattern and to avoid certification issues, they want to have the possibility to write to a single node and to load balance the reads. The application is not able to do the Read/Write splitting so, without a tool to do the splitting, only one node can be used for all the traffic. Of course, to make things easy, they use a lot of Java code that set tons of sessions variables. Furthermore, for ISO 27001 compliance, they want to be able to log all the queries for security analysis (and also for performance analysis, why not?). So, high query rate, Read/Write splitting and full query logging, like I said a challenging problem. We experimented with a few solutions. One was a hardware load balancer that failed miserably – the implementation was just too simple, using only regular expressions. Another solution we tried was ScaleArc but it needed many rules to whitelist the set session variables and to repeat them to multiple servers. ScaleArc could have done the job but all the rules increases the CPU load and the cost is per CPU. The queries could have been sent to rsyslog and aggregated for analysis. Finally, the HA implementation is rather minimalist and we had some issues with it. Then, we tried MaxScale. At the time, it was not GA and was (is still) young. Nevertheless, I wrote a query logging filter module to send all the queries to a Kafka cluster and we gave it a try. Kafka is extremely well suited to record a large flow of queries like that. In fact, at 30k qps, the 3 Kafka nodes are barely moving with cpu under 5% of one core. Although we encountered some issues, remember MaxScale is very young, it appeared to be the solution with the best potential and so we moved forward. The folks at MariaDB behind MaxScale have been very responsive to the problems we encountered and we finally got to a very usable point and the test in the pilot environment was successful. The solution is now been deployed in the staging environment and if all goes well, it will be in production soon. The following figure is simplified view of the internals of MaxScale as configured for the customer: The blocks in the figure are nearly all defined in the configuration file. We define a TCP listener using the MySQL protocol (client side) which is linked with a router, either the readwritesplit router or the readconn router. The first step when routing a query is to assign the backends. This is where the read/write splitting decision is made. Also, as part of the steps required to route a query, 2 filters are called, regexp (optional) and Genlog. The regexp filter may be used to hot patch a query and the Genlog filter is the logging filter I wrote for them. The Genlog filter will send a json string containing about what can be found in the MySQL general query log plus the execution time. Authentication attempts are also logged but the process is not illustrated in the figure. A key point to note, the authentication information is cached by MaxScale and is refreshed upon authentication failure, the refresh process is throttled to avoid overloading the backend servers. The servers are continuously monitored, the interval is adjustable, and the server status are used when the decision to assign a backend for a query is done. In term of HA, I wrote a simple Pacemaker resource agent for MaxScale that does a few fancy things like load balancing with IPTables (I’ll talk about that in future post). With Pacemaker, we have a full fledge HA solution with quorum and fencing on which we can rely. Performance wise, it is very good – a single core in a virtual environment was able to read/write split and log to Kafka about 10k queries per second. Although MaxScale supports multiple threads, we are still using a single thread per process, simply because it yields a slightly higher throughput and the custom Pacemaker agent deals with the use of a clone set of MaxScale instances. Remember we started early using MaxScale and the beta versions were not dealing gracefully with threads so we built around multiple single threaded instances. So, since a conclusion is needed, MaxScale has proven to be a very useful and flexible tool that allows to elaborate solutions to problems that were very hard to tackle before. In particular, if you need to perform read/write splitting, then, try MaxScale, it is best solution for that purpose I have found so far. Keep in touch, I’ll surely write other posts about MaxScale in the near future.