Troy Hunt (@troyhunt, blog) had a great, very hands-on 2-day workshop about webapp security at NDC Oslo. Here are my notes. Highlights – resources Personal security and privacy https://www.entropay.com/ – a Prepaid Virtual Visa Card mailinator.com – tmp email f-secure VPN https://www.netsparker.com/ – scan a site for issues (insecure cookies, framework disclosure, SQL injection, …) (lot of $k) Site security https://report-uri.io/ – get reports when CSP rules violated; also displays CSP headers for a site in a human-friendly way https://securityheaders.io/ check quality of headers wrt security free SSL – http://www.startssl.com/, https://www.cloudflare.com/ (also provides web app firewall and other protections) ; SSL quality check: https://www.ssllabs.com/ssltest/ https://letsencrypt.org/ – free, automated, open Certificate Authority (Linux Found., Mozilla) Breaches etc. http://arstechnica.com/security/2015/06/hack-of-cloud-based-lastpass-exposes-encrypted-master-passwords/ https://twitter.com/jmgosney – one of ppl behind http://passwordscon.org . http://password-hashing.net experts panel. Team Hashcat. http://arstechnica.com/security/2012/12/25-gpu-cluster-cracks-every-standard-windows-password-in-6-hours/ To follow ! http://krebsonsecurity.com/ ! http://www.troyhunt.com/ ! https://www.schneier.com/ ! https://twitter.com/mikko (of F-Secure) also great [TED] talks kevin mitnick (jailed for hacking; twitter, books) Books http://www.amazon.com/We-Are-Anonymous-LulzSec-Insurgency/dp/0316213527 – easy read, hard to put down http://www.amazon.com/Ghost-Wires-Adventures-Worlds-Wanted/dp/1441793755 – about Mitnick’s hacking, social engineering, living on the run ? http://www.amazon.com/Art-Intrusion-Exploits-Intruders-Deceivers/dp/0471782661/ Mitnick: http://www.amazon.com/Art-Deception-Controlling-Element-Security/dp/076454280X/ – social engineering Other https://www.xssposed.org/ See https://www.drupal.org/SA-CORE-2014-005 https://www.youtube.com/watch?v=Qvhdz8yE_po – Havij example http://www.troyhunt.com/2013/07/everything-you-wanted-to-know-about-sql.html, http://www.troyhunt.com/2010/05/owasp-top-10-for-net-developers-part-1.html, http://www.troyhunt.com/2012/12/stored-procedures-and-orms-wont-save.html, Googlee: find config files with SA access info: `inurl:ftp inurl:web.config filetype:config sa` https://scotthelme.co.uk/hardening-your-http-response-headers/ and https://securityheaders.io/ https://developer.mozilla.org/en-US/docs/Web/Security/Public_Key_Pinning – prevent MITM wappalyzer chrome plugin displaying info about the server and client that can be detected (jQuery, NewRelic, IIS, win OS, …) http://www.troyhunt.com/2015/05/do-you-really-want-bank-grade-security.html http://www.troyhunt.com/2012/05/everything-you-ever-wanted-to-know.html tool: https://github.com/gentilkiwi/mimikatz extract plaintexts passwords, hash, PIN code and kerberos tickets from memory on Windows Notes HackYourselfFirst.troyhunt.com – an example app with many vulnerabilities Note: maximizing your browser window will share info about your screen size, which might help to identify you haveibeenpwned.com – Troy’s online DB of hacked accounts Tips check robots.txt to know what to access Example Issues no https on login page insecure psw requirements cookies not secure flag => sent over http incl. AuthCookie) psw sent in clear text in confirm email user enumeration, f.eks. an issue with AdultFriendFinder – entry someone’s email to login to find out whether they’ve an account post illegal chars, get them displayed => injection no anti-automation (captcha) login confirm. email & autom. creating 1m accounts => sending 1m emails => pisses ppl off, likely increase one’s spam reputation (=> harder to send emails) brute-force protection? ### XSS Reflected XSS: display unescaped user input Encoding context: HTML, JS, CSS … have diff. escape sequences for the same char (e.g. <) – look at where they’re mixed Check the encoding consistency – manual encoding, omitting some chars JS => load ext resources, access cookies, manipulate the DOM Task: stal authCookie via search ### SQL injection Error-based injection: when the DB helps us by telling us what is wrong -> use ti learn more and even show some data Ex.: http://hackyourselffirst.troyhunt.com/Make/10?orderby=supercarid <—— supercarid is a column name orderby=(select * from userprofile) … learn about DB sructure, force an exception that shows the valueex.: (select top 1 cast(password) as int from userprofile) => “Conversion failed for the nvar value ‘passw0rd …’” Tips think of SQL commands that disclose structure: sys.(tables,columns), system commands enumerate records: nest queries: select top X ows asc then top 1 rows from that desc write out how you think the query works / is being constructed internally cast things to invalid types to disclose values in err msgs (or implicit cast due to -1 ..) #### Defenses whitelist input data types (id=123 => onlyallow ints) enumerable values – check against an appropr. whitelist if the value is stored – who uses it, how? making query/insertion safe permissions: give read-only permissions as much as possible; don’t use admin user from your webapp ### Mobile apps Look at HTTP req for sensitive data – creds, account, … Apps may ignore certificate validations In your app: param tampering, auth bypass, direct object refs Weak often: airlines, small scale shops, fast foods, … Tips certificate pining – the app has the fingerprint of the server cert. hardcoded and doesn’t trust even “valid” MITM certificate (banks, dropbox, …)x ### CSRF Cross-Site Request Forgery = make the user send a request => their auth cookie included async Ajax req to another site forbidden but that doesn’t apply to normal post Protection anti-forgery tags ### Understanding fwrk disclosure http://www.shodanhq.com/ -> search for “drupal 7” -> pwn How disclosed: headers familiar signs – jsessionid cookie for java, … The default error and 404 responses may help to recognize the fwr HTML code (reactid), “.do” for Sttruts implicit: order of headers (Apache x IIS), paths (capitalized?), response to improper HTTP version/protocol, => likely still possible to figure out the stack but not possible to simple search for fwrk+version ### Session hijacking Steal authentication cookie => use for illegal requests. Persistence over HTTP of auth., session: cookie, URL (but URL insecure – can be shared) Session/auth ID retrieval: insecure transport, referrer, stored in exceptions, XSS Factors limiting hijacking: short duration expiry, keyed to client device / IP (but IPs may rotate, esp, on mobile devices => be very cautious) DAY 2 ——– ### Cracking passwords Password hashing: salt: so that 2 ppl choosing the same psw will have a different hash => cracking is # salts * # passwords inst. of just N has cracking tips: character space Dictionary: passw0rd, … Mutations: manipulation and subst. of characters Tips: 1Password , LastPass, …. GPU ~ 100* faster than CPU #### Ex: Crack with hashcat common psw dict + md5-hashed passwords => crack ./hashcat-cli64.bin –hash-type=0 StratforHashes.txt hashkiller.com.dic # 23M psw dict -> Recovered.: 44 326/860 160 hashes [obs duplications] in 4 min (speed 135.35k plains) Q: What dictionary we use? Do we apply any mutations to it? ### Account enumeration = Does XY have an account? Multiple vectors (psw reset, register a new user with the same e-mail, …) Anti-automation: is there any? It may be inconsistent across vectors Does it matter? (<> privacy needs) How to “ask” the site and how to identify + and – responses? Timing attacks: distinguish positive x negative response based on the latency differing between the two ### HTTPS Confidentiality, Integrity, Authenticity Traffic hijacking: [a href="https://www.wifipineapple.com/"]https://www.wifipineapple.com/ – wifi hotspot with evil capabilities monitor probe requests (the phone looks for networks it knows), present yourself as one of those, the phone connects autom. (if no encryption) Consider everything sent over HTTP to be compromised Look at HTTPS content embedded in untrusted pages (iframes, links) – e.g. payment page embedded in http Links HSTS Preload – tell Chrome, FF that your site should only be ever loaded over HTTPS – https://hstspreload.appspot.com/ https://www.owasp.org/index.php/HTTP_Strict_Transport_Security header ### Content Scurity Policy header https://developer.chrome.com/extensions/contentSecurityPolicy See e.g. https://haveibeenpwned.com/ headers w/o CSP anything can be added to the page via a reflected XSS risk Anyth, can be added to the DOM downstream (on a proxy) With CSP the browser will only load resources you white-list; any violations can be reported Use e.g. https://report-uri.io/home/generate to create it and the report to watch for violations to fine tune it. ### SQL injection cont’d (Yesterday: Error-Based) #### Union Based SQLi Modify the query to union whatever other data and show them. More data faster than error-based inj. Ex.: http://hackyourselffirst.troyhunt.com/CarsByCylinders?Cylinders=V12 : V12 -> `V12′ union select voteid, comments collate SQL_Latin1_General_CP1_CI_AS from vote– ` #### Blind Boolean (laborious) Blind inj.: We can’t always rely on data being explicitly returned to the UI => ask a question, draw a conclusion about the data. Ex: http://hackyourselffirst.troyhunt.com/Supercar/Leaderboard?orderBy=PowerKw&asc=false -> ordedby => case when (select count(*) from userprofile) > 1 then powerkw else topspeedkm end Extract email: Is ascii of the lowercase char #1 < ascii of m ? Automation: SqlMap #### Time based blind injection When no useful output returned but yes/no responses differ significantly in how much time they take. F.ex. ask the db to delay the OK response. MS SQL: IF ‘b’ > ‘a’ WAITFOR DELAY ’00:00:05′ ### Brute force attacks Are there any defences? Often not How are defences impl? block the req resources block the src IP rate limit (by src IP) ### Automation penetration testing apps and services such as Netsparker, WhiteHatSec targets identification: shodan, googledorks, randowm crawling think aout the actions that adhere to a pattern – sql injection, fuzzing (repeat a req. trying diff. values for fields – SQLi, …), directory enumeration automation can be used for good – test your site tip: have autom. penetration testing (and perhaps static code analysis) as a part fo your build pipeline Task: Get DB schema using sqlmap (see python2.7 sqlmap.py –help) ### Protection Intrusion Detection System (IDS) – e.g. Snort Web Application Firewall (WAF) – e.g. CloudFare ($20/m)

One of the most important kinds of data a Platform as a Service (PaaS) can leverage is its knowledge of guest applications that run within its purview. A PaaS should know all sorts of things about guest applications – their architecture, dependencies, scale across infrastructure, and more. Data including application resource utilization metrics (CPU, RAM, etc.) are key for things like data center capacity planning, policy enforcement, and application isolation in the enterprise. A PaaS such as Apprenda provides this information through a centralized single lens – in our case, a collection of RESTful APIs – making it easier than ever before to run analytics on application metrics in the data center. Apprenda’s approach as a PaaS is to provide developers and platform operators with helpful information through platform extensibility and APIs. This is because there are plenty of tools in the data center that provide advanced analytic capabilities, so long as you can feed them the information they need. We integrate with tools like System Center, New Relic, and more all the time because these are the tools our customers have invested in, and they are great at what they do. Our job is not to reinvent these tools but instead to provide data. Apprenda captures information about applications such as their duration of deployment, resource policy (allocation of CPU and memory), actual utilization of resources, scale (# of instances), custom metadata, and more. All of this information can be fed into data center tools that help IT make important, data-driven, decisions. In the land of DevOps, however, it is not uncommon for folks to use this data in creative and innovative ways. Often times this means using the mechanism “du jour,” which can be scripting (PowerShell), a programming language (R), or an entire runtime (Node.js) to quickly and effectively grab, process, and manipulate data. In a big example, let’s look at R, which is a powerful programming language centered on data mining and statistical analysis. It provides straightforward facilities for many types of data-analytics techniques, and is extensible using community maintained packages. In the simple example below, I use standard R functions plus three packages (easily included using R’s install.packages() function): 1. jsonlite for parsing JSON data that the Apprenda API returns. 2. httr for handling the HTTP requests necessary to authenticate and retrieve data. 3. plotrix for help rendering a plot of retrieved data. From there it’s pretty straightforward. The first step is to authenticate with your Apprenda environment: I’ve now stored my Apprenda session token in a variable called ‘token.’ I’ll include that token as a header in my API call to get application data: GET() is a function provided by the httr package that simplifies an HTTP request to the API. I’ve added the Apprenda session token to the HTTP Headers for authentication, and included a query string parameter that will help return all currently running application workloads on the platform. The data that is returned is parsed and stored in the variable (in R, a vector) called ‘r’ which now has 151 records, one for each application workload. Each record in ‘r’ has 15 variables (properties) that we can use to run analytics across the entire collection of results. For the purposes of illustration, I’m going to use the variable componentType, which represents Apprenda’s knowledge of the type of application workload that was deployed – there are seven self-explanatory types: UserInterface, PublicUserInterface,WindowsService, JavaWebApplication, LinuxService, WcfService, and Database. When the collection is then grouped bycomponentType, it becomes pretty simple to plot a chart showing the distribution of workload by the type of component: The resulting plot (pie3D() comes from the plotrix package) looks like this: I’ve had conversations with IT folks who couldn’t describe the architectural makeup of their application portfolio in any level of detail, yet in this case we pulled the data in real time with one line of R. Admittedly, a pie chart is a pretty watered down way to look at this information, but the point is the data is available and can be grouped, filtered, manipulated, and analyzed very simply with R. For this example, I used the open-source edition of RStudio. Some other powerful information that could be gleaned from the platform’s APIs: 1. The average discrepancy between resource allocation and actual utilization per workload. (This is helpful in capacity planning.) 2. The longest -running application workload. 3. The most distributed applications. (This could aid in scaling decisions.) There are many more. A PaaS such as Apprenda is, by nature, in a unique spot in the data center stack because it maintains knowledge of both infrastructure and applications. It also serves as a hub for data that, when analyzed creatively, provides new insights. These insights are an opportunity for enterprises to enhance their practices to better serve developers and applications while operating more efficiently than ever.

In this article, we go over several commands developers and data scientists can use to create data frames using existing data frames.

It’s been a year since I started the quest for a highly-effective Data Knowledge Stack and the Hibernate Master Class contains over fifty articles already. Now that I covered many aspects of database transactions, JDBC and Java Persistence, it’s time to assemble all the pieces together into the High-Performance Java Persistence book. An Agile publishing experience Writing a book is a very time-consuming and stressful process and the last thing I needed was a very tight schedule. After reading Antonio Goncalves’s story, I chose the self-publishing way. In the end, I settled for Leanpub because it allows me to publish the book incrementally. This leads to a better engagement with readers, allowing me adapt the book content on the way. The content At its core, the book is about getting the most out of your persistence layer and that can only happen when your application resonates with the database system. Because concurrency is inherent to database processing, transactions play a very important role in this regard. The first part will be about some basic performance-related database concepts such as: locking, batching, connection pooling. In the second part, I will explain how an ORM can actually improve DML performance. This part will include the Hibernate Master Class findings. The third part is about advance querying techniques with jOOQ. If you enjoy reading this article, you might want to subscribe to my newsletter and get a discount for my book as well. Get involved The Agile methodologies are not just for software development. Writing a book in a Lean style can shorten the feed-back period and readers can get involved on the way. If you have any specific request or you are interested in this project, you can join my newsletter and follow my progress. Buy it! The book is 100% done, and you can check out the full Table of Content onLeanpub. If you enjoyed this article, I bet you are going to love my book as well. The ebook The PDF, ePUB and Kindle (MOBI) versions can be bought on Leanpub. The print version The print version is available on Amazon, Amazon.co.uk, Amazon.de or Amazon.fr. Presentations If you are not convinced, then check out the following two presentations: High-Performance JDBC from Voxxed Days Bucharest High-Performance Hibernate from JavaZone

[This article was written by Wyatt Lindsay] New Relic’s Community Forum is designed to be a place for our users to share their experiences, questions, problems, and fixes. The collective expertise and creativity of the New Relic community has generated some outstanding solutions to everyday issues, and we want to call out some of them in the area of .NET agent deployment excellence. Basic installation Installing New Relic’s .NET agent is designed to be simple: run the installer on the target host and choose the features you want to include. For Microsoft Azuredeployments, install one of our NuGet packages. Installation requires a reset of IIS for the software to load into your application. To upgrade, we recommend first stopping IIS, installing the newer agent version, and then starting up IIS again. It’s also possible to perform a “silent” (manual) install using msiexec.exe, for example: msiexec.exe /i C:\NewRelicAgent.msi /qb NR_LICENSE_KEY= INSTALLLEVEL=1 See the documentation for complete manual installation options. Leveraging PowerShell Scripting the silent installation provides convenience and flexibility. Here’s an example script provided by community member Jon Carl in this post: $msiName = $licenseKey = $arguments = "/i $msiName /L*v install.log /qn NR_LICENSE_KEY=$licenseKey" if ($msiName -ne $null) { $exitCode = (Start-Process -FilePath "msiexec" -ArgumentList $arguments -Wait -PassThru).ExitCode; if($exitCode -eq 0) { Write-Host "Installation successful!" -ForegroundColor Green } else { Write-Host "Installation unsuccessful. Exitcode: $exitCode" -ForegroundColor Red } } This script works great when run directly on the target machine. Another forum user (Kym McGain) noticed that the installation didn’t complete before the session ended when executing the script remotely. This caused the installer to quit partway through. Kym posted this script that uses a ‘while’ loop to ensure the installer completes. As a bonus, it stops IIS before and restarts it after the software installs. As mentioned above, these steps are usually needed when upgrading. $installNewRelic = { $runProcess = { param($process,$arguments) $res = Start-Process -FilePath $process -ArgumentList $arguments -Wait -PassThru while ($res.HasExited -eq $false) { Write-Host "Waiting for $process..." Start-Sleep -s 1 } $exitCode = $res.ExitCode if($exitCode -eq 0) { Write-Host "$process successful!" -ForegroundColor Green } else { Write-Host "$process unsuccessful. Exitcode: $exitCode" -ForegroundColor Red } } $msiName = $licenseKey = $arguments = "/i $msiName /L*v install.log /qn NR_LICENSE_KEY=$licenseKey" Invoke-Command $runProcess -ArgumentList "IISRESET","/STOP" Invoke-Command $runProcess -ArgumentList "msiexec.exe",$arguments Invoke-Command $runProcess -ArgumentList "IISRESET","/START" } Chef, Puppet, and Chocolatey Deployment options abound for modern Web developers. These solutions often require a known download path and installer name. New Relic offers a consistent filepath and MSI name for the agent in an effort to make automated deployment easier for .NET customers: http://download.newrelic.com/dot_net_agent/release/NewRelicDotNetAgent_x64.msi http://download.newrelic.com/dot_net_agent/release/NewRelicDotNetAgent_x86.msi Several Community members have created packages for these utilities. Chocolatey users are invited to use the following NuGet package created by kireevco: https://chocolatey.org/packages/newrelic-dotnet New Relic community member ePitty built a Puppet module to handle .NET agent deployment: https://github.com/epitty1023/puppet-newrelicappmon Chef users, meanwhile, can check out the following cookbook for .NET and many other platforms New Relic supports: http://community.opscode.com/cookbooks/newrelic New Relic Community member E_Bow wrote a Chef recipe that goes a step further by stopping IIS before the installation and starting it again after completion: #Stop IIS iis_site 'Website' do action [:stop] end # install latest Newrelic agent from web include_recipe 'newrelic::repository' include_recipe node['newrelic']['dotnet-agent']['dotnet_recipe'] license = node['newrelic']['application_monitoring']['license'] windows_package 'Install New Relic .NET Agent' do source node['newrelic']['dotnet-agent']['https_download'] options "/qb NR_LICENSE_KEY=#{license} INSTALLLEVEL=#{node['newrelic']['dotnet-agent']['install_level']}" installer_type :msi action :install end #Start IIS iis_site 'Website' do action [:start] end The author states that they were unable to pull the New Relic license key from the configuration JSON in Chef Overrides, requiring them to modify the config file on each machine and manually enter the key. We invite any Chef experts out there to extend and improve this recipe so that it correctly pulls the license key. We are continually impressed by the smarts and spirit of our New Relic Community Forum members, and jump at the chance to highlight their contributions. Look for more Forum projects in the New Relic blog in the future. Do you have your own approach, tips, or recipes? Please share them in the New Relic Community Forum.

[This article was written by Matt Kiernan] We are excited to announce that Logentries’ new SQL-like query language, LEQL, is now available for more advanced analytics and easy extraction of valuable insights from your log data. A SQL-Like Query Language If you’ve ever used SQL, LEQL should feel familiar. In fact, Logentries already supports a number of SQL-like search functions, including: SUM: Sums a set of values COUNT: Counts the number of times a value occurs GROUPBY: Groups values by a unique key UNIQUE: Enables the count of only unique values With the rollout of LEQL, we’ll be introducing four new query functions: MIN: Calculate the minimum value of a specified key MAX: Calculate the maximum value of a specified key SORT: Display results sorted either ascending or descending TIMESLICE: Specifies how to group by time (e.g. by specific number of minutes, hours or days) A Consistent Yet Expressive Syntax We believe a reliable query language depends on a consistently enforced syntax. For this reason, we’ll be enforcing how queries are structured. Here’s an example of how an old query would change with LEQL: Old pages>0 | GroupBY(dbName) | SUM(pages) New where(pages>0) groupby (dbName) calculate(SUM:pages) *In this example, pages & dbName are Key names in log events Notice how the search logic gets wrapped in a where() clause, used for refining your search to return only results that match your search criteria (i.e. where events include the text or Key “pages”.) groupBy() is an optional clause that enables you to organize your search results into groups by specifying a Key from a Key-Value Pair (i.e. key: value). Calculations made within your query get utilized in the calculate()clause. When building your query, you no longer need to separate sections with pipes “|”. Though we believe in the value of a consistent query syntax, we also believe in the importance of giving users an expressive language that is easy to use and delivers expected results. We’re taking the following steps to make LEQL easy to use: Outdated saved queries will automatically be converted into LEQL – no effort required where clauses will automatically be added to any new query you write LEQL terms will not be case sensitive An updated search bar will provide a query builder and validator An updated search bar & query builder An Updated Search Bar & Query Builder As we rollout LEQL, we’ll be introducing a new search bar, allowing users to switch between a simple & advanced modes based on their preference. Simple mode “Simple mode” provides an easy way to build queries by providing a list of the available functions. Type-Assist will show a list of keys to associate with each functions, or new keys can be typed manually. Advanced mode “Advanced mode” will allow users to type their queries manually. Type-Assist will autocomplete key names while the new search bar will automatically validate query syntax. July 1st Rollout The LEQL rollout will take place in phases, beginning July 1st and will continue over the next few weeks to update all plans. If you’d like early beta access to LEQL, or have any questions, feel free to reach us at [email protected].

Written by Yves Trudeau Ever since MySQL replication has existed, people have dreamed of a good solution to automatically split read from write operations, sending the writes to the MySQL master and load balancing the reads over a set of MySQL slaves. While if at first it seems easy to solve, the reality is far more complex. First, the tool needs to make sure it parses and analyses correctly all the forms of SQL MySQL supports in order to sort writes from reads, something that is not as easy as it seems. Second, it needs to take into account if a session is in a transaction or not. While in a transaction, the default transaction isolation level in InnoDB, Repeatable-read, and the MVCC framework insure that you’ll get a consistent view for the duration of the transaction. That means all statements executed inside a transaction must run on the master but, when the transaction commits or rollbacks, the following select statements on the session can be again load balanced to the slaves, if the session is in autocommit mode of course. Then, what do you do with sessions that set variables? Do you restrict those sessions to the master or you replay them to the slave? If you replay the set variable commands, you need to associate the client connection to a set of MySQL backend connections, made of at least a master and a slave. What about temporary objects like with “create temporary table…”? How do you deal when a slave lags behind or what if worse, replication is broken? Those are just a few of the challenges you face when you want to build a tool to perform read/write splitting. Over the last few years, a few products have tried to tackle the read/write split challenge. The MySQL_proxy was the first attempt I am aware of at solving this problem but it ended up with many limitations. ScaleARC does a much better job and is very usable but it stills has some limitations. The latest contender is MaxScale from MariaDB and this post is a road story of my first implementation of MaxScale for a customer. Let me first introduce what is MaxScale exactly. MaxScale is an open source project, developed by MariaDB, that aims to be a modular proxy for MySQL. Most of the functionality in MaxScale is implemented as modules, which includes for example, modules for the MySQL protocol, client side and server side. Other families of available modules are routers, monitors and filters. Routers are used to determine where to send a query, Read/Write splitting is accomplished by the readwritesplit router. The readwritesplit router uses an embedded MySQL server to parse the queries… quite clever and hard to beat in term of query parsing. There are other routers available, the readconnrouter is basically a round-robin load balancer with optional weights, the schemarouter is a way to shard your data by schema and the binlog router is useful to manage a large number of slaves (have a look at Booking.com’s Jean-François Gagné’s talk at PLMCE15 to see how it can be used). Monitors are modules that maintain information about the backend MySQL servers. There are monitors for a replicating setup, for Galera and for NDB cluster. Finally, the filters are modules that can be inserted in the software stack to manipulate the queries and the resultsets. All those modules have well defined APIs and thus, writing a custom module is rather easy, even for a non-developer like me, basic C skills are needed though. All event handling in MaxScale uses epoll and it supports multiple threads. Over the last few months I worked with a customer having a challenging problem. On a PXC cluster, they have more than 30k queries/s and because of their write pattern and to avoid certification issues, they want to have the possibility to write to a single node and to load balance the reads. The application is not able to do the Read/Write splitting so, without a tool to do the splitting, only one node can be used for all the traffic. Of course, to make things easy, they use a lot of Java code that set tons of sessions variables. Furthermore, for ISO 27001 compliance, they want to be able to log all the queries for security analysis (and also for performance analysis, why not?). So, high query rate, Read/Write splitting and full query logging, like I said a challenging problem. We experimented with a few solutions. One was a hardware load balancer that failed miserably – the implementation was just too simple, using only regular expressions. Another solution we tried was ScaleArc but it needed many rules to whitelist the set session variables and to repeat them to multiple servers. ScaleArc could have done the job but all the rules increases the CPU load and the cost is per CPU. The queries could have been sent to rsyslog and aggregated for analysis. Finally, the HA implementation is rather minimalist and we had some issues with it. Then, we tried MaxScale. At the time, it was not GA and was (is still) young. Nevertheless, I wrote a query logging filter module to send all the queries to a Kafka cluster and we gave it a try. Kafka is extremely well suited to record a large flow of queries like that. In fact, at 30k qps, the 3 Kafka nodes are barely moving with cpu under 5% of one core. Although we encountered some issues, remember MaxScale is very young, it appeared to be the solution with the best potential and so we moved forward. The folks at MariaDB behind MaxScale have been very responsive to the problems we encountered and we finally got to a very usable point and the test in the pilot environment was successful. The solution is now been deployed in the staging environment and if all goes well, it will be in production soon. The following figure is simplified view of the internals of MaxScale as configured for the customer: The blocks in the figure are nearly all defined in the configuration file. We define a TCP listener using the MySQL protocol (client side) which is linked with a router, either the readwritesplit router or the readconn router. The first step when routing a query is to assign the backends. This is where the read/write splitting decision is made. Also, as part of the steps required to route a query, 2 filters are called, regexp (optional) and Genlog. The regexp filter may be used to hot patch a query and the Genlog filter is the logging filter I wrote for them. The Genlog filter will send a json string containing about what can be found in the MySQL general query log plus the execution time. Authentication attempts are also logged but the process is not illustrated in the figure. A key point to note, the authentication information is cached by MaxScale and is refreshed upon authentication failure, the refresh process is throttled to avoid overloading the backend servers. The servers are continuously monitored, the interval is adjustable, and the server status are used when the decision to assign a backend for a query is done. In term of HA, I wrote a simple Pacemaker resource agent for MaxScale that does a few fancy things like load balancing with IPTables (I’ll talk about that in future post). With Pacemaker, we have a full fledge HA solution with quorum and fencing on which we can rely. Performance wise, it is very good – a single core in a virtual environment was able to read/write split and log to Kafka about 10k queries per second. Although MaxScale supports multiple threads, we are still using a single thread per process, simply because it yields a slightly higher throughput and the custom Pacemaker agent deals with the use of a clone set of MaxScale instances. Remember we started early using MaxScale and the beta versions were not dealing gracefully with threads so we built around multiple single threaded instances. So, since a conclusion is needed, MaxScale has proven to be a very useful and flexible tool that allows to elaborate solutions to problems that were very hard to tackle before. In particular, if you need to perform read/write splitting, then, try MaxScale, it is best solution for that purpose I have found so far. Keep in touch, I’ll surely write other posts about MaxScale in the near future.

I have project where I need to output some reports as CSV-files. I found a good library called CsvHelper from NuGet and it works perfect for me. After some playing with it I was able to generate CSV-files that were shown correctly in Excel. Here is some sample code and also extensions that make it easier to work with DataTables. Simple report Here’s the simple fragment of code that illustrates how to use CsvHelper. using (var writer = new StreamWriter(Response.OutputStream)) using (var csvWriter = new CsvWriter(writer)) { csvWriter.Configuration.Delimiter = ";"; csvWriter.WriteField("Task No"); csvWriter.WriteField("Customer"); csvWriter.WriteField("Title"); csvWriter.WriteField("Manager"); csvWriter.NextRecord(); foreach (var project in data) { csvWriter.WriteField(project.Code); csvWriter.WriteField(project.CustomerName); csvWriter.WriteField(project.Name); csvWriter.WriteField(project.ProjectManagerName); csvWriter.NextRecord(); } } Of course, you can use other methods to output whole object or object list with one shot. I just needed here custom headers that doesn’t match property names 1:1. Generic helper for DataTable Some of my projects come from service layer as DataTable. I don’t want to add new models or Data Transfer Objects (DTO) with no good reason and DataTable is actually flexible enough if you need to add new fields to report and you want to do it fast. As DataTables are not supported by default (yet?), I wrote simple extension methods that work on DataTable views. When called on DataTable it selects default view automatically. The idea is – you can set filter on default data view and leave out the rows you don’t need. If you just want to show DataTable to screen as table then check out my posting Simple view to display contents of DataTable. public static class CsvHelperExtensions { public static void WriteDataTable(this CsvWriter csvWriter, DataTable table) { WriteDataView(csvWriter, table.DefaultView); } public static void WriteDataView(this CsvWriter csvWriter, DataView view) { foreach (DataColumn col in view.Table.Columns) { csvWriter.WriteField(col.ColumnName); } csvWriter.NextRecord(); foreach (DataRowView row in view) { foreach (DataColumn col in view.Table.Columns) { csvWriter.WriteField(row[col.ColumnName]); } csvWriter.NextRecord(); } } } And here is simple MVC controller action that gets data as DataTable and returns it as CSV-file. The result is CSV-file that opens correctly in Excel. [HttpPost] public void ExportIncomesReport() { var data = // Get DataTable here Response.ContentType = "text/csv"; Response.AddHeader("Content-disposition", "attachment;filename=IncomesReport.csv"); var preamble = Encoding.UTF8.GetPreamble(); Response.OutputStream.Write(preamble, 0, preamble.Length); using (var writer = new StreamWriter(Response.OutputStream)) using (var csvWriter = new CsvWriter(writer)) { csvWriter.Configuration.Delimiter = ";"; csvWriter.WriteDataTable(data); } } One thing to notice – with CsvHelper we have full control over a stream where we write data and this way we can write more performant code. Related Posts .Net Framework 4.0: string.IsNullOrWhiteSpace() method Exporting GridView Data to Excel Code Contracts: Hiding ContractException How to dump object properties My object to object mapper source released The post Generating CSV-files on .NET appeared first on Gunnar Peipman - Programming Blog.

In my continued playing around with web logs in R I wanted to process the logs for a day and see what the most popular URIs were. I first read in all the lines using the read_lines function in readr and put the vector it produced into a data frame so I could process it using dplyr. library(readr) dlines = data.frame(column = read_lines("~/projects/logs/2015-06-18-22-docs")) In the previous post I showed some code to extract the URI from a log line. I extracted this code out into a function and adapted it so that I could pass in a list of values instead of a single value: extract_uri = function(log) { parts = str_extract_all(log, "\"[^\"]*\"") return(lapply(parts, function(p) str_match(p[1], "GET (.*) HTTP")[2] %>% as.character)) } Next I ran the following function to count the number of times each URI appeared in the logs: library(dplyr) pages_viewed = dlines %>% mutate(uri = extract_uri(column)) %>% count(uri) %>% arrange(desc(n)) This crashed my R process with the following error message: segfault cause 'memory not mapped' I narrowed it down to a problem when doing a group by operation on the ‘uri’ field and came across this post which suggested that it was handled more cleanly in more recently version of dplyr. I upgraded to 0.4.2 and tried again: ## Error in eval(expr, envir, enclos): cannot group column uri, of class 'list' That makes more sense. We’re probably returning a list from extract_uri rather than a vector which would fit nicely back into the data frame. That’s fixed easily enough by unlisting the result: extract_uri = function(log) { parts = str_extract_all(log, "\"[^\"]*\"") return(unlist(lapply(parts, function(p) str_match(p[1], "GET (.*) HTTP")[2] %>% as.character))) } And now when we run the count function it’s happy again, good times!

I’ve been playing around with some web logs using R and I wanted to extract everything that existed in double quotes within a logged entry. This is an example of a log entry that I want to parse: log = '2015-06-18-22:277:548311224723746831\t2015-06-18T22:00:11\t2015-06-18T22:00:05Z\t93317114\tip-127-0-0-1\t127.0.0.5\tUser\tNotice\tneo4j.com.access.log\t127.0.0.3 - - [18/Jun/2015:22:00:11 +0000] "GET /docs/stable/query-updating.html HTTP/1.1" 304 0 "http://neo4j.com/docs/stable/cypher-introduction.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36"' And I want to extract these 3 things: /docs/stable/query-updating.html http://neo4j.com/docs/stable/cypher-introduction.html Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36 i.e. the URI, the referrer and browser details. I’ll be using the stringr library which seems to work quite well for this type of work. To extract these values we need to find all the occurrences of double quotes and get the text inside those quotes. We might start by using the str_match function: > library(stringr) > str_match(log, "\"[^\"]*\"") [,1] [1,] "\"GET /docs/stable/query-updating.html HTTP/1.1\"" Unfortunately that only picked up the first occurrence of the pattern so we’ve got the URI but not the referrer or browser details. I tried str_extract with similar results before I found str_extract_all which does the job: > str_extract_all(log, "\"[^\"]*\"") [[1]] [1] "\"GET /docs/stable/query-updating.html HTTP/1.1\"" [2] "\"http://neo4j.com/docs/stable/cypher-introduction.html\"" [3] "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36\"" We still need to do a bit of cleanup to get rid of the ‘GET’ and ‘HTTP/1.1′ in the URI and the quotes in all of them: parts = str_extract_all(log, "\"[^\"]*\"")[[1]] uri = str_match(parts[1], "GET (.*) HTTP")[2] referer = str_match(parts[2], "\"(.*)\"")[2] browser = str_match(parts[3], "\"(.*)\"")[2] > uri [1] "/docs/stable/query-updating.html" > referer [1] "https://www.google.com/" > browser [1] "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36" We could then go on to split out the browser string into its sub components but that’ll do for now!

Survey of open source PostgreSQL users found adoption continues to rise with 55% of users deploying it for mission-critical applications Bedford, MA – June 23, 2015 – EnterpriseDB (EDB), the leading provider of enterprise-class Postgres products and database compatibility solutions, today announced the results of its “PostgreSQL Adoption Survey 2015,” a biennial survey of open source PostgreSQL users. Conducted by EnterpriseDB, the survey found PostgreSQL adoption continuing to rise, with 55% of users – up from 40% two years ago – deploying it for mission-critical applications and 77% of users are dedicating all new application deployments to PostgreSQL. These findings give voice to end users and confirm such industry indicators as increasing job listings and monthly rankings on DB-Engines that have pointed to rising interest in and demand for PostgreSQL, also called Postgres. The growing popularity of Postgres also comes as traditional software vendors suffer setbacks in the marketplace. The enterprise-class performance, security and stability of Postgres, on par with traditional database vendors for most corporate workloads, meanwhile have helped position Postgres among the solutions from the world’s largest vendors. The opportunity to transform their data center economics has helped fuel downloads of Postgres as well. End users reported cutting costs with Postgres, with 41% reporting they had first-year cost savings of 50% or more. They’re using Postgres to build web 2.0 applications using unstructured data as evidenced by the 64% of respondents who said they were working with JSON/JSONB and the 47% who said they were using Postgres for collaboration applications. “Postgres is empowering organizations to transform the economics of IT. IT can invest in the customer engagement applications that differentiate their operations from their competition instead of continuing to pay the steep and rising licensing and support fees charged by traditional database vendors,” said Marc Linster, senior vice president of products and services of EnterpriseDB. “With the expanding adoption, EnterpriseDB has experienced dramatic growth year over year, providing the software, services and support that organizations need to be successful with Postgres.” Database Migrations, Replacements The findings also support statements in a recent Gartner report that reflect the widespread acceptance of open source databases. “By 2018, more than 70% of new in-house applications will be developed on an OSDBMS, and 50% of existing commercial RDBMS instances will have been converted or will be in process,” according to the April 2015 Gartner report, The State of Open-Source RDBMs, 2015.* Among Postgres users, the survey findings show migrations are already under way with 37% reporting they had migrated applications from Oracle or Microsoft SQL Server to Postgres. Many users were still planning further migrations, with 37% of PostgreSQL users saying they will gradually replace their legacy systems with Postgres, compared to 29% who said that in the 2013 survey. Further, end users predict their deployments of Postgres will expand significantly, with 32% saying they anticipate production deployments of Postgres to increase by at least 50% over the next year. The survey, conducted by EnterpriseDB using an online tool in May 2015, queried registered users of PostgreSQL and drew 274 respondents worldwide from government organizations and companies ranging in size and industry. *The State of Open-Source RDBMs, 2015, by Donald Feinberg and Merv Adrian, published on April 21, 2015. Connect with EnterpriseDB Read the blog: http://blogs.enterprisedb.com/ Follow us on Twitter: http://www.twitter.com/enterprisedb Become a fan on Facebook: http://www.facebook.com/EnterpriseDB?ref=ts Join us on Google+: https://plus.google.com/108046988421677398468 Connect on LinkedIn: http://www.linkedin.com/company/enterprisedb

What's New in this Release? Aspose team is pleased to announce the release of Aspose.Pdf for Java 10.3.0. It provides better license initialization capabilities. As shared in earlier blogs, we introduced a method clear() in com.aspose.pdf.MemoryCleaner class, which provides Memory Cleanup features so that memory is set free from unused objects. This method optimizes API performance as system resources are released, leaving API with sample resources to perform various PDF creation and manipulation operations. In this new release, we have also optimized TextStamp operation. Other than these improvements, a better support for UTF8 and UTF16 characters is provided, when converting TEXT files to PDF format. Cross file format conversions are one of the salient features offered by our API. Therefore, the PDF to HTML, the PDF to DOC, transformation of PDF pages to Image format as well as the Image to PDF conversion features are specifically improved. Among these features, the text manipulation is also improved while searching and replacing TextFragments inside the PDF file. Starting this new release, we are providing a single code base (.jar) file targeting JDK 1.6 and its compatible with JDK 1.6, 1.7 and later versions. Some important improved features included in this release are given below Increase TextStamp creation performance com.aspose.pdf.MemoryCleaner.clear() method nulls the license object as well Aspose.Pdf 9.5.2 to HTML conversion issue on particular file UTF-8 characters not appearing properly License implementation difference in 9.3.0 and 10.2.0 with Java web application java.awt.HeadlessException in Headless Mode PDF to Image - Conversion process stucks in infinite loop Text to PDF: Incorrect rendering of UTF8 text in output PDF Text to PDF: Incorrect rendering of UTF16 text in output PDF gets wrong coordinates of seached Text Image to PDF: API throws IllegalArgumentException PDF to PNG - Process hangs during conversion PDF to HTML: text is distorted in output HTML PDF to DOC: Text renders incorrectly Image to PDF throws IllegalArgumentException exception PDF to HTML - StringIndexOutOfBoundsException being generated PDF to Image - conversion method stuck and never returns Hyperlink text/contents are not visible in PDF file Overview: Aspose.Pdf for Java Aspose.Pdf is a Java PDF component to create PDF documents without using Adobe Acrobat. It supports Floating box, PDF form field, PDF attachments, security, Foot note & end note, Multiple columns document, Table of Contents, List of Tables, Nested tables, Rich text format, images, hyperlinks, JavaScript, annotation, bookmarks, headers, footers and many more. Now you can create PDF by API, XML and XSL-FO files. It also enables you to converting HTML, XSL-FO and Excel files into PDF. Homepage of Aspose.Pdf for Java Download Aspose.Pdf for Java

Last year some data from the Citizens Advise Bureau revealed that 7 out of 10 potentially successful employment cases are not being pursued, with a good 50 percent of those being down to financial issues. Whilst it’s tempting to think that we are all equal in front of the law, there remains a distinct sense that we are anything but. It’s a major reason why companies such as Logikcull are trying to make the whole process easier and more efficient. It’s believed that the e-discovery process can contribute to around 70 percent of the costs of any legal proceeding, so reducing the time involved in that can be a huge cost saver. Using the crowd Other organizations are attempting to make the legal process more affordable by recruiting the crowd to help meet the legal costs involved. For instance, I wrote about LexShares towards the end of last year, who are a kind of crowd based investment site. You can ‘invest’ in a particular case, thus giving the plaintiff funds to pursue their case. If the case is successful, the backer gets their money back plus a bit of the damages. If the case fails, then they lose their money. Another crowd based venture launched in the UK recently. The site, called CrowdJustice, aims to provide funding to cases that would normally struggle to do so. Supporting public interest cases The site was founded by Julia Salasky, who previously worked for the UN, and aims to specialize in so called public interest cases. “CrowdJustice allows communities to band together to access the courts to protect their communal assets – like their local hospital – or shared values – like human rights. Successive governments have made access to justice harder and more expensive but we are using the power of the crowd to try and stem the tide,” she says. She suggests that cuts to legal aid has made it harder for poorer people to access adequate legal protection, especially when it comes to challenging large institutions. This is especially so when the end game doesn’t necessarily result in a large payout. This could include, for instance, the destruction of a local bird sanctuary or even much larger issues such as torture. Despite effecting huge numbers of people, it is often very difficult for communities to channel their energies towards fighting the case collectively. As such, these kind of cases typically require a determined individual to pursue the cause on their own. The hope is that the CrowdJustice platform will make this considerably easier. Whether it’s CrowdJustice or LexStorm or Logikcull, there are certainly a wide range of projects aiming to change the legal industry for the better. It will be fascinating to watch them as they unfold and witness the impact they have. Original post

Written by Barry Jones. PostgreSQL is becoming the relational database of choice for web development for a whole host of good reasons. That means that development teams have to make a decision on whether to host their own or use a database as a service provider. The two biggest players in the world of PostgreSQL are Heroku PostgreSQL and Amazon RDS for PostgreSQL. Today I’m going to compare both platforms. Heroku was the first big provider to make a push for PostgreSQL instead of MySQL for application development. They launched their Heroku PostgreSQL platform back in 2007. Amazon Web Services first announced their RDS for PostgreSQL service in November 2013 during the AWS re:Invent conference to an overwhelming ovation by the programmers in attendance. Pricing Comparison Before I get too far into the features, let’s cover the pricing differences up front. Of course, both services have areas with different value propositions for productivity and maintenance that go beyond these direct costs. However, it’s worth it to understand the basic costs so you can weigh those values against your needs later. Heroku PostgreSQL has the simplest pricing. The rates and what you get for them are very clearly set at a simple per-month rate that includes the database, storage, data transfer, I/O, backups, SLA, and any other features built into the pricing tier. With RDS for PostgreSQL, pricing is broken down into smaller units of individual resource usage. That means there are more factors involved in estimating the price, so it’s a little tougher to draw an exact comparison to Heroku PostgreSQL. You have the price per hour for the instance type, higher if it’s a multiple availability zone instance, cheaper if you pay an upfront cost to reserve the instance for one to three years; storage cost and storage class (both single and multi AZ); provisioned IOPs rate; backup storage, and data transfer… then there are a whole lot of special cases to consider. Also, keep in mind that you get one year free of the cheapest plan when you sign up. Here is a comparison of an RDS plan to the Heroku Premium 4 plan: Heroku Premium 4 $1,200 / Month 15 GB RAM 512 GB storage 500 connections High Availability Max 15 minutes downtime/month 1 week rollback Point in time recovery Encryption at rest Continuous protection (offsite Write-Ahead-Log) RDS for PostgreSQL $1,156/month on demand or $756/month 1 year reserved db.m3.xlarge Multi-AZ at $0.780/hr ($580) 4 vCPU, 15GB RAM Encryption at rest 512 GB provisioned (SSD) at $0.250/GB ($128) 2000 provisioned IOPS at $0.20/IOPS ($400) Estimated backup storage in excess of free for 1 week rollback, 512 GB at $0.095/GB ($48) Data transfer estimated at $0 for most use cases 22 minutes downtime/month (based on AWS RDS SLA 99.95% uptime) Now, here are the caveats with such a comparison: Heroku isn’t disclosing the number of CPUs associated with their plans. Heroku’s High Availability is equivalent to AWS RDS Multi-AZ. In both setups, a read replica is maintained in a different geographic region specifically for the purpose of automatic failover in the event of an outage. With Heroku, your storage is fully allocated, and you do not pay for IOPS. As such, we don’t know what the limits are for IOPS, but they are very high performance databases. I allocated the minimum IOPS that AWS would allow for 512 GB, which was 2,000. We could go as high as 5,000 IOPS which would increase the price by $600/month. The AWS RDS backups may cost nothing depending on how much of the provisioned storage is actually being used. Backup storage is free up to the level of provisioned storage, and backups are generally smaller, incremental, and do not include the significant space used by indexes. This estimate was based on the seven days of storage needed to allow for one week rollback. AWS RDS storage can be scaled up on the fly, so your specific needs for RAM versus storage could create a wildly different pricing pattern. This comparison is aiming to draw an equivalent. AWS only charges for data transfers out of your availability zone (not including multi-AZ transfers), so transfer rates will not apply in most cases. Clear as mud. Setup Complexity Heroku PostgreSQL setup is dead simple. Whenever you create a PostgreSQL project, a free dev plan is already created with it with a connection waiting. Upgrading the database simply gives you a new connection string with a set username, password, hostname, and database identifier that are all randomly generated by their system. The database connection must be secure but is accessible anywhere on the internet, including directly from your home computer. You can also choose whether to deploy it in the US East region or in the European region. RDS for PostgreSQL setup is slightly more involved; you must select the various options outlined in the pricing section, including the instance type, whether or not it should be Multi-AZ, whether to enable encryption at rest, type of storage, how much to provision, IOPs to provision (if any), backup retention period, whether or not to enable automatic minor version upgrades, selection of backup and maintenance windows, database identifier, name, port, master user and password, which availability zone you want it to be created in and the selection of your VPC group and subnet group, and your database configuration. Obviously, RDS gives you significantly more control over the details. Depending on your point of view, that could be good or bad. The database configuration, for example, has a set of defaults for each database version for each instance type. You can take these defaults and make modifications to them with your own custom settings and then save those as your own parameter group to assign to this and any future databases that you may choose to create. The initial setup time can be slightly more involved because of the various factors like VPC, subnet groups, and public accessibility. However, once these have been defined the first time for your account, everything gets much closer to a point-and-click experience. Host Locations, Regional Restrictions Heroku operates with the AWS US East Region (us-east-1) and Europe (eu-west-1). This also means that your database will be restricted to these regions. Availability Zones are managed internally. If you choose to use Heroku PostgreSQL with something hosted in a different AWS region than those two, you should expect more latency between database requests and transfer rates may apply. Likewise, if you wish to use AWS RDS for PostgreSQL with a Heroku application, just ensure that it is set up in the appropriate region. Security and Access Considerations Within Heroku PostgreSQL, you’re given a randomized username with a randomized password and a randomized database name that must be connected to over SSL. Their network (as well as Amazon’s) have built-in protections against scanners that could potentially brute-force access such a database. That is fairly secure. The downside is that anybody who needs access to the database and has the connection information can do so from anywhere in the world. This is more of a Human Resources-level risk from departed programmers on a project than anything else, but it is something to be aware of nonetheless. Swapping out the database credentials after having a programmer leave the team will generally alleviate this concern. On the other hand, AWS RDS for PostgreSQL has a much more comprehensive security policy. The ability to set and define a VPC and private subnet groups will allow you to restrict database access to only the servers and people who need it. You have the ability to create as many database users with various permission levels as you like in order to more easily manage multiple users or applications accessing the database with different permission levels, while providing a log trail. Thanks to VPC, even if somebody did have the connection information, they still couldn’t access the database without being able to get inside the VPC. For stricter (although more complex) security, RDS wins hands down. Depending on complexity, team, and the development state of your application, this level of security paranoia may not yet make sense and could be more of a headache than you want to manage. You can also configure it with the same access rules used by Heroku PostgreSQL. Backup/Restore/Upgrade Both platforms offer very similar options for backup and restore. Both have scheduled backups, point-in-time recovery, restoration to a new copy, and the ability to create snapshots. Upgrades are more involved. On both platforms, major version upgrades will involve some downtime, which can’t be avoided. Heroku provided three options that all involve some manual steps to complete: copying data, promoting an upgraded follower, or using the pg:upgrade command for an in-place upgrade of larger databases. The pg:upgrade most closely resembles the upgrade process on RDS. With RDS, you select the Modify option for your instance and change the version. It will create pre- and post-snapshots around the in-place upgrade while maintaining the exact same connection string. RDS will allow you to schedule the database upgrade automatically within your set maintenance window. Heroku PostgreSQL will automatically apply minor upgrades and security patches, while RDS allows you to choose whether or not you want them to do that automatically within your maintenance window. Both are fairly straightforward processes, although the RDS process is a little more hands-off in this case. Feature/Extension Availability As of this writing, AWS RDS for PostgreSQL has version 9.3.1–9.3.6 and 9.4.1, while Heroku PostgreSQL has 9.1, 9.2, 9.3, and 9.4. Minor version upgrades are automatic with Heroku, so the point releases are unnecessary. Heroku PostgreSQL has been around longer and because of that has more legacy versions available for their existing users. RDS launched with 9.3 and does not appear to have any intention to support older versions. In addition to all of the functionality built into PostgreSQL, there’s a constantly growing set of extensions. Both platforms have these extensions in common: hstore citext ltree isn cube dict_int unaccent PostGIS dblink earthdistance fuzzystrmatch intarray pg_stat_statements pgcrypto pg_trgm tablefunc uuid-ossp pgrowlocks btree_gist PL/pgSQL PL/Tcl PL/Perl PL/V8 Available on Heroku PostgreSQL: pgstattuple Available on AWS RDS for PostgreSQL: postgres_fdw chkpass intagg tsearch2 sslinfo Here are the full lists for both Heroku PostgreSQL and AWS RDS for PostgreSQL. Scaling Options “Scaling” is a tricky word with databases because it means different things depending on the needs of your application. Scaling for writes vs. reads is based on low intensity and high volume (web traffic) compared to low volume and high intensity (analytics). The most common scaling case on the web is scaling for read traffic. Both Heroku and RDS address this need with the ability to create read replicas. RDS calls them read replicas and Heroku calls them followers, but they’re essentially the same thing: a copy of the database, receiving live updates via the write-ahead-log over the wire to allow you to spread read traffic over multiple servers. This is commonly referred to as horizontal scaling. To create read replicas on either platform is a point-and-click operation. Vertical scaling refers to increasing or decreasing the power of the hardware of your database in place. AWS and Heroku each handle this scenario differently. Heroku instructs users to create a follower of the newly desired database class and then promote it to the primary database once it’s caught up, destroying the original afterwards. Your application will need to update its database connection information to use the new database. If your RDS database is a multi-AZ database, then the failover database will be upgraded first. Once ready, the connection will automatically failover to that instance while the primary is then upgraded, switching back to the primary afterwards. Without a Multi-AZ, you can do the upgrade in place, but downtime will vary depending on the size of the database. Your other option is to create a read replica with the newly desired stats and then promote it to primary when it is ready, just as Heroku recommends. To scale beyond the standard vertical and horizontal options for something that can handle distributed write scaling, neither option is a particularly good fit. It will probably be necessary to either manage your own Postgres-XC installation or restructure your application to isolate the write-heavy traffic into a more use-case specific data source. Monitoring AWS RDS for PostgreSQL comes with all of the standard AWS monitoring options via Cloudwatch. Cloudwatch provides extensive metrics that you can track history with a granular ability to set up alerts via email or SNS notifications (basically webhooks). These are great for integrating with tools like PagerDuty. Heroku PostgreSQL monitoring relies more on logs and command line tools. Their pgextras command line tool will show current information about what’s going on in the database, including bloat, blocking queries, cache and index hit ratios, identification of unused indexes, and the ability to kill specific queries. These tools, while not involving the stat tracking over time that you get from Cloudwatch, provide extremely valuable insights into what’s going on with your database that you don’t come close to getting from RDS. You can see more examples of pg-extras on GitHub. These type of insights are invaluable in tuning your application and database to avoid the problems you’d need a monitor to catch in the first place. Other historical data is available in the logs, although Heroku recommends trying out Librato (which can work with any PostgreSQL database but has a Heroku plugin available for automatic configuration). Additionally, free New Relic plans will provide a wealth of insight into what’s going on with your application and database. While Cloudwatch provides more detailed insight as to what’s going on within the machine, Heroku uses the metrics seen within pg-extras to monitor and notify you of the various problems they see that require correction on your end. If data corruption happens, Heroku identifies and fixes it. Security problems, they’ll handle it. A DBA or a DevOps position will care significantly more about the Cloudwatch metrics. Heroku PostgreSQL tries to focus on making sure you don’t have to worry about it. Dataclips One bonus feature that you get from Heroku PostgreSQL is Dataclips. Dataclips are basically a method for storing and sharing read-only queries among your team for the sake of reporting without having to grant access to every person who may need to see them. Just type in a query and view the results right there on the page. The queries are version controlled; if your team is passing them around and tweaking them, you’ll be able to see the changes over time. In my personal experience, I’ve found dataclips to be a lifesaver, specifically for working with non-programmer teams. When business or support staff need information on sales, fraud, user behavior, account activity, or anything else we happen to have in there, I’ve always had the ability to write up a query to get at the information. Before dataclips, this meant that I needed to write up the query, save it somewhere, usually export the result set to a CSV or spreadsheet, and then email it to whomever was requesting it. Eventually, this becomes a routine activity that you’re having to handle at every request. Enter dataclips. Now I can take that query and just send the random hashed link over to whoever requested the information. If they want more up-to-date information the next day, week, or month, they need only refresh the page. I write the query, then never hear that request again. That is a developer time-saver right there. You can save them and name them, as well as manage more strict access if need be. Summary and Recommendation Overall, AWS RDS for PostgreSQL will usually be cheaper and more tightly tailorable to exactly what your application’s needs are. You’ll have much more granular control over access, security, monitoring, alerts, geographic location, and maintenance plans. With Heroku PostgreSQL, you’ll pay a little bit more on a simplified pricing structure, although all of your development databases will be free. You won’t be able to control a lot of the details that RDS gives you access to, but that’s partially by design so that you don’t have to deal with managing those details. With Heroku, you’ll get insights directly into how your database is performing and using the internal resources to help you catch, tune, and improve your setup before it becomes a problem. If I had to choose, I’d probably go with Heroku and Heroku PostgreSQL as a startup while I focused on actually getting my application developed and getting customers in the door. The value proposition of saving time to focus on business goals so we can build a revenue stream would be of the greatest importance. Then when things grew to a point that the database was no longer changing as much, it might make sense to start migrating things over to RDS as we focus on locking things down to focus on stability, long-term maintenance, and security. In the end, it really boils down to what costs you more: time or infrastructure. If time costs you more, go with Heroku PostgreSQL. If infrastructure costs you more, go with RDS. Having both platforms living within the AWS datacenters makes switching between the two a lot easier as your needs change.

I don't often write about tools I use when for my daily software development tasks. I recently realized that I really should start to share more often my workflows and weapons of choice. One thing that I have a hard time enduring while doing Python code reviews, is people writing utility code that is not directly tied to the core of their business. This looks to me as wasted time maintaining code that should be reused from elsewhere. So today I'd like to start with retrying, a Python package that you can use to… retry anything. It's OK to fail Often in computing, you have to deal with external resources. That means accessing resources you don't control. Resources that can fail, become flapping, unreachable or unavailable. Most applications don't deal with that at all, and explode in flight, leaving a skeptical user in front of the computer. A lot of software engineers refuse to deal with failure, and don't bother handling this kind of scenario in their code. In the best case, applications usually handle simply the case where the external reached system is out of order. They log something, and inform the user that it should try again later. In this cloud computing area, we tend to design software components with service-oriented architecture in mind. That means having a lot of different services talking to each others over the network. And we all know that networks tend to fail, and distributed systems too. Writing software with failing being part of normal operation is a terrific idea. Retrying In order to help applications with the handling of these potential failures, you need a plan. Leaving to the user the burden to "try again later" is rarely a good choice. Therefore, most of the time you want your application to retry. Retrying an action is a full strategy on its own, with a lot of options. You can retry only on certain condition, and with the number of tries based on time (e.g. every second), based on a number of tentative (e.g. retry 3 times and abort), based on the problem encountered, or even on all of those. For all of that, I use the retrying library that you can retrieve easily on PyPI. retrying provides a decorator called retry that you can use on top of any function or method in Python to make it retry in case of failure. By default, retry calls your function endlessly until it returns rather than raising an error. import randomfrom retrying import retry @retrydef pick_one(): if random.randint(0, 10) != 1: raise Exception("1 was not picked") This will execute the function pick_one until 1 is returned by random.randint. retry accepts a few arguments, such as the minimum and maximum delays to use, which also can be randomized. Randomizing delay is a good strategy to avoid detectable pattern or congestion. But more over, it supports exponential delay, which can be used to implement exponential backoff, a good solution for retrying tasks while really avoiding congestion. It's especially handy for background tasks. @retry(wait_exponential_multiplier=1000, wait_exponential_max=10000)def wait_exponential_1000(): print "Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards" raise Exception("Retry!") You can mix that with a maximum delay, which can give you a good strategy to retry for a while, and then fail anyway: # Stop retrying after 30 seconds anyway>>> @retry(wait_exponential_multiplier=1000, wait_exponential_max=10000, stop_max_delay=30000)... def wait_exponential_1000():... print "Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards"... raise Exception("Retry!")...>>> wait_exponential_1000()Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwardsWait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwardsWait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwardsWait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwardsWait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwardsWait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwardsTraceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/local/lib/python2.7/site-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/local/lib/python2.7/site-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/local/lib/python2.7/site-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "", line 4, in wait_exponential_1000 Exception: Retry! A pattern I use very often, is the ability to retry only based on some exception type. You can specify a function to filter out exception you want to ignore or the one you want to use to retry. def retry_on_ioerror(exc): return isinstance(exc, IOError) @retry(retry_on_exception=retry_on_ioerror)def read_file(): with open("myfile", "r") as f: return f.read() retry will call the function passed as retry_on_exception with the exception raised as first argument. It's up to the function to then return a boolean indicating if a retry should be performed or not. In the example above, this will only retry to read the file if an IOError occurs; if any other exception type is raised, no retry will be performed. The same pattern can be implemented using the keyword argument retry_on_result, where you can provide a function that analyses the result and retry based on it. def retry_if_file_empty(result): return len(result) <= 0 @retry(retry_on_result=retry_if_file_empty)def read_file(): with open("myfile", "r") as f: return f.read() This example will read the file until it stops being empty. If the file does not exist, an IOError is raised, and the default behavior which triggers retry on all exceptions kicks-in – the retry is therefore performed. That's it! retry is really a good and small library that you should leverage rather than implementing your own half-baked solution!

I’m still working my way through the exercises in Think Bayes and in Chapter 6 needed to do some cleaning of the data in a CSV file containing information about the Price is Right. I downloaded the file using wget: wget http://www.greenteapress.com/thinkbayes/showcases.2011.csv And then loaded it into R and explored the first few rows using dplyr library(dplyr) df2011 = read.csv("~/projects/rLearning/showcases.2011.csv") > df2011 %>% head(10) X Sep..19 Sep..20 Sep..21 Sep..22 Sep..23 Sep..26 Sep..27 Sep..28 Sep..29 Sep..30 Oct..3 1 5631K 5632K 5633K 5634K 5635K 5641K 5642K 5643K 5644K 5645K 5681K 2 3 Showcase 1 50969 21901 32815 44432 24273 30554 20963 28941 25851 28800 37703 4 Showcase 2 45429 34061 53186 31428 22320 24337 41373 45437 41125 36319 38752 5 ... As you can see, we have some empty rows which we want to get rid of to ease future processing. I couldn’t find an easy way to filter those out but what we can do instead is have empty columns converted to ‘NA’ and then filter those. First we need to tell read.csv to treat empty columns as NA: df2011 = read.csv("~/projects/rLearning/showcases.2011.csv", na.strings = c("", "NA")) And now we can filter them out using na.omit: df2011 = df2011 %>% na.omit() > df2011 %>% head(5) X Sep..19 Sep..20 Sep..21 Sep..22 Sep..23 Sep..26 Sep..27 Sep..28 Sep..29 Sep..30 Oct..3 3 Showcase 1 50969 21901 32815 44432 24273 30554 20963 28941 25851 28800 37703 4 Showcase 2 45429 34061 53186 31428 22320 24337 41373 45437 41125 36319 38752 6 Bid 1 42000 14000 32000 27000 18750 27222 25000 35000 22500 21300 21567 7 Bid 2 34000 59900 45000 38000 23000 18525 32000 45000 32000 27500 23800 9 Difference 1 8969 7901 815 17432 5523 3332 -4037 -6059 3351 7500 16136 ... Much better!

Need to replace all occurances of a pattern text and replace it with a captured group? Something like this in Java works nicely: String html = "myurl\n" + "myurl2\n" + "myurl3"; html = html.replaceAll("id=(\\w+)'?", "productId=$1'"); Here I swapped the query name from "id" to "productId" on all the links that matched my criteria. But what happen if I needed to pre-process the captured ID value before replacing it? Let's say now I want to do a lookup and transform the ID value to something else? This extra requirement would lead us to dig deeper into Java RegEx package. Here is what I come up with: import java.util.regex.*; ... public String replaceAndLookupIds(String html) { StringBuffer newHtml = new StringBuffer(); Pattern p = Pattern.compile("id=(\\w+)'?"); Matcher m = p.matcher(html); while (m.find()) { String id= m.group(1); String newId = lookup(id); String rep = "productId=" + newId + "'"; m.appendReplacement(newHtml, rep); } m.appendTail(newHtml); return newHtml.toString(); }

ThreadLocal is a simple way to have per-thread data that cannot be accessed concurrently by other threads, without requiring great effort or design compromises.

PART 1 > THREADS - Top 80 interview questions and answers (detailed explanation with programs) Question 61. class MyRunnable implements Runnable{ public void run(){ for(int i=0;i<3;i++){ System.out.println("i="+i+" ,ThreadName="+Thread.currentThread().getName()); } } } public class MyClass { public static void main(String...args){ MyRunnable runnable=new MyRunnable(); System.out.println("start main() method"); Thread thread1=new Thread(runnable); Thread thread2=new Thread(runnable); thread1.start(); thread2.start(); System.out.println("end main() method"); } } Answer. Thread behaviour is unpredictable because execution of Threads depends on Thread scheduler, start main() method will be the printed first, but after that we cannot guarantee the order of thread1, thread2 and main thread they might run simultaneously or sequentially, so order of end main() method will not be guaranteed. /*OUTPUT start main() method end main() method i=0 ,ThreadName=Thread-0 i=0 ,ThreadName=Thread-1 i=1 ,ThreadName=Thread-0 i=2 ,ThreadName=Thread-0 i=1 ,ThreadName=Thread-1 i=2 ,ThreadName=Thread-1 */ Question 62. class MyRunnable implements Runnable{ public void run(){ for(int i=0;i<3;i++){ System.out.println("i="+i+" ,ThreadName="+Thread.currentThread().getName()); } } } public class MyClass { public static void main(String...args) throws InterruptedException{ System.out.println("In main() method"); MyRunnable runnable=new MyRunnable(); Thread thread1=new Thread(runnable); Thread thread2=new Thread(runnable); thread1.start(); thread1.join(); thread2.start(); thread2.join(); System.out.println("end main() method"); } } Answer. We use join() methodto ensure all threads that started from main must end in order in which they started and also main should end in last. In other words join() method waited for this thread to die. /*OUTPUT In main() method i=0 ,ThreadName=Thread-0 i=1 ,ThreadName=Thread-0 i=2 ,ThreadName=Thread-0 i=0 ,ThreadName=Thread-1 i=1 ,ThreadName=Thread-1 i=2 ,ThreadName=Thread-1 end main() method */ Question 63. class MyRunnable implements Runnable { public void run() { try { while (!Thread.currentThread().isInterrupted()) { Thread.sleep(1000); System.out.println("x"); } } catch (InterruptedException e) { System.out.println(Thread.currentThread().getName() + " ENDED"); } } } public class MyClass { public static void main(String args[]) throws Exception { MyRunnable obj = new MyRunnable(); Thread t = new Thread(obj, "Thread-1"); t.start(); System.out.println("press enter"); System.in.read(); t.interrupt(); } } Answer. "press enter" will be printed first then thread1 will keep on printing x until enter is pressed, once enter is pressed "Thread-1 ENDED" will be printed. System.in.read() causes main thread to go from running to waiting state (thread waits for user input) /* OUTPUT press enter x x x x Thread-1 ENDED */ Question 64. class MyRunnable implements Runnable{ public void run(){ synchronized (this) { System.out.println("1 "); try { this.wait(); System.out.println("2 "); } catch (InterruptedException e) { e.printStackTrace(); } } } } public class MyClass { public static void main(String[] args) { MyRunnable myRunnable=new MyRunnable(); Thread thread1=new Thread(myRunnable,"Thread-1"); thread1.start(); } } Answer. Thread acquires lock on myRunnable object so 1 was printed but notify wasn't called so 2 will never be printed, this is called frozen process. Deadlock is formed, These type of deadlocksare called Frozen processes. /*OUTPUT 1 */ Question 65. import java.util.ArrayList; /* Producer is producing, Producer will allow consumer to * consume only when 10 products have been produced (i.e. when production is over). */ class Producer implements Runnable{ ArrayList sharedQueue; Producer(){ sharedQueue=new ArrayList(); } @Override public void run(){ synchronized (this) { for(int i=1;i<=3;i++){ //Producer will produce 10 products sharedQueue.add(i); System.out.println("Producer is still Producing, Produced : "+i); try{ Thread.sleep(1000); }catch(InterruptedException e){e.printStackTrace();} } System.out.println("Production is over, consumer can consume."); this.notify(); } } } class Consumer extends Thread{ Producer prod; Consumer(Producer obj){ prod=obj; } public void run(){ synchronized (this.prod) { System.out.println("Consumer waiting for production to get over."); try{ this.prod.wait(); }catch(InterruptedException e){e.printStackTrace();} } int productSize=this.prod.sharedQueue.size(); for(int i=0;i Q61- Q80

In my previous blog post I described how I’d refactored some scraping code I’ve been working on to use iterators and ended up with a function which returned a generator containing all the events for one BBC live text match: match_id = "32683310" events = extract_events("data/raw/%s" % (match_id)) >>> print type(events) The next thing I wanted to do is get the events for multiple matches which meant I needed to glue together multiple generators into one big generator. itertools’ chain function does exactly what we want: itertools.chain(*iterables) Make an iterator that returns elements from the first iterable until it is exhausted, then proceeds to the next iterable, until all of the iterables are exhausted. Used for treating consecutive sequences as a single sequence. First let’s try it out on a collection of range generators: import itertools gens = [(n*2 for n in range(0, 3)), (n*2 for n in range(4,7))] >>> gens [ at 0x10ff3b140>, at 0x10ff7d870>] output = itertools.chain() for gen in gens: output = itertools.chain(output, gen) Now if we iterate through ‘output’ we’d expect to see the multiples of 2 up to and including 12: >>> for item in output: ... print item ... 0 2 4 8 10 12 Exactly as we expected! Our scraping code looks like this once we plug the chaining in: matches = ["32683310", "32683303", "32384894", "31816155"] raw_events = itertools.chain() for match_id in matches: raw_events = itertools.chain(raw_events, extract_events("data/raw/%s" % (match_id))) ‘raw_events’ now contains a single generator that we can iterate through and process the events for all matches.