DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

The Latest Testing, Deployment, and Maintenance Topics

article thumbnail
Geek Reading Week of June 26, 2015
Leading today, VentureBeat reports on the quiet launch of Google’s Cloud Source Repositories. This seems like something we should have heard more about, but I don’t remember seeing anything about it. Amazon AWS announces the availability of all things Alexa, the Skills Kit, the Voice Service and a Fund. Last but not least, we have AJ Kohn, from Blind Five Year Old, talking about click-through rate being a ranking signal on Google’s search results. I don’t talk about SEO much, but reading AJ’s work is always fascinating. As always, enjoy today’s items, and please participate in the discussions on these sites. Top Stories Google has quietly launched a GitHub competitor, Cloud Source Repositories | VentureBeat Alexa Skills Kit, Alexa Voice Service, Alexa Fund | AWS Official Blog Startups, Career and Process Why offices are where work goes to die | Swizec Teller Unleashing the power of small teams | Andreas Papathanasis What Is A Tester? | Developsense Blog What happens when you stop relying on resumes | Aline Lerner Design and Development Swift 2: SIMD | Russ Bishop Why is Git better than Mercurial? | Javalobby Create a Maven archetype | Javalobby pip -t: A simple and transparent alternative to virtualenv | Zoomer Analytics Killing Off Wasabi – Part 1 | Fog Creek Blog WebAssembly- Explained | Modus Create Generating JSON Schema from XSD with JAXB and Jackson | Inspired by Actual Events AI, Machine Learning, Research and Advanced Algorithms Applying Machine Learning to Text Mining with Amazon S3 and RapidMiner | Amazon AWS Big Data, Visualization, SQL and NoSQL Is Click Through Rate A Ranking Signal? | Blind Five Year Old Cache-friendly binary search | Bannalia Discovering the Computer Science Behind Postgres Indexes | Java Code Geeks How an open-source competitive benchmark helped to improve databases | ArangoDB Security, Encryption and Cryptography Cracking JXcore… Again | Mark Haase Link Collections The Daily Six Pack: June 25, 2015 | Dirk Strauss Double Shot #1517 | A Fresh Cup Dew Drop – June 25, 2015 (#2042) | Morning Dew The Daily Six Pack: June 26, 2015 | Dirk Strauss
June 27, 2015
by Robert Diana
· 989 Views
article thumbnail
What Different Security Testing Methodologies Are Out There?
Every business has unique characteristics that set it apart from other organizations, even within the same industry. For this reason, it shouldn't be surprising that there's not a one-size-fits-all approach to app security testing. Each company has certain protection expectations and regulations to adhere to, making it essential to find the best way to achieve these goals. Here are a few examples of security testing methodologies available for quality assurance teams to leverage: Black box With black box testing, QA professionals put themselves in the shoes of the hacker and attempt to break the app through various attack vectors. This methodology can yield a lot of information and help better secure the program from actual threats. A white paper by Security Innovation noted that software testers first analyze the system's architecture and business model to identify any security vulnerabilities. Looking over the software logic in this way can uncover subtle security and privacy issues that may not have been noticed otherwise, such as defects in design, input, system dependency, authentication, cryptography and information disclosure. "Although white box code inspection is good for analyzing static behavior, only black box exploratory testing can determine the dynamic behavior of how a system is implemented and used, the coupling between systems and the interactions of the distributed systems," Security Innovation wrote. Dynamic For QA teams that like to execute code, dynamic testing is the approach for them. This methodology checks the running application for how it behaves and responds to a variety of inputs. This is done to ensure that the product meets up with established regulations and is giving the expected outcomes. IBM noted that dynamic analysis is especially useful to identify code coverage, as it can discover bugs in paths that have gone untested. While dynamic testing can be manual work for testers, it can also yield significant information that will help mitigate defects and produce quality products. Static In contrast to dynamic testing, static approaches directly review the source code, often through an automated test management solution. TechTarget contributor Michael Cobb noted that this methodology occurs at the implementation phase, rather than when the app is running, and often helps mitigate vulnerabilities involved with industry compliance standards. Automation in this area can reduce the amount of time it takes to complete these tasks. However, it may not be able to detect sophisticated threats, which can be supplemented by dynamic security testing. "A thorough source code review has an advantage over dynamic testing," Cobb wrote. "Nothing is hidden from analysts during a source code review, so they can examine exactly how data flows through a program. By solving the problem at the code level, static testing reduces the number of security-related design and coding defects, and the severity of any defects that make it through to the release version, thus dramatically improving the overall security of the application." There are a number of security testing methodologies that organizations can pursue, and designing a combination of approaches may result with a solution that's best for their needs. Using these strategies, companies can better ensure the protection of sensitive information while providing users with the software testing tools needed to succeed.
June 27, 2015
by Sanjay Zalavadia
· 1,050 Views
article thumbnail
Managing 673 Maven Projects with POM Explorer
When a team works with a lot of maven projects it becomes quickly painful to do some basic tasks like: manage versionning and connections between the different projects. releasing and opening versions, especially when the maven-release plugin needs to be run on many projects and when versionning is not standard. managing external dependencies also can become complex, and ensuring that a single version of a dependency is used accross different projects is sometimes not a trivial question to ask. in a one word, applying transformations on a dependency graph is difficult. mind-mapping the dependency graph is difficult when the number of projects grows. That can increase the amount of time needed by new people to understand a project graph, and that also makes maintaining and changing things difficult. checking consistency and optimizing the dependency graph is not an easy task neither. having an always up-to-date build of snapshots and release is not easy when projects are distributed everywhere. Pom Explorer So there is the Pom Explorer tool which tries to address those problems by providing those functionalities : release a graph : release a pom or all poms and its/theirs dependencies and updates all dependent poms accross multiple repositories and projects, change a gav : updates a project’s gav and make all the project which depends on it follow this change. manages properties, dependency management, and so on. Pom-explorer knows what pom.xml to update and where to update. If a dependency specifies ${foobar.version}, pom-explorer will go to update the foobar.version property. query the dependency graph to retrieve pertinent information about your projects, statistics and check functions are also available, display 3d interactive graph, export graphml files, find not used dependencies and other similar problems, list java classes provided by artifacts, list java classes referenced by artifacts, runs a light and efficient web server so local and shared usage is possible. The tool will also support automatically building projects in order to always have such or such project always up to date. Use cases In this article, I will show some common use cases possible with this tool. Installation First one needs to install and run the software. Put yourself in a temporary directory and type those commands : git clone https://github.com/ltearno/pom-explorer.git cd pom-explorer java -jar target/pom-explorer.jar The program should welcome you and ask you to go to this address : http://localhost:90 This is the console to the application. You can type commands in the prompt, they will be sent to the server and it will answer. You can use up and down arrows to recall past commands. Let’s start by typing ? to get the available commands : Analyze of repositories OK. First we will analyse a directory where there are many maven projects, then we will work a bit to optimize those projects. You will have to adapt the exercise to your computer. Let’s analyse my git repositories directory : analyze directory c:\documents\repos This will analyze my projects and construct an in-memory graph of the dependency graph : Now, the program knows about everything on my projects, let’s start asking questions ! List of GAVs… Let’s get the list of all existing GAVs (groupId, artifactId, version) in the graph. There will be my projects and all the GAVs on which they depend. Type this command : gav li Note that you can type only the first letters of a command, as long as there is no ambiguity. Here li stands for list. Find dependencies on an obsolete artifact As I look through the list of GAVs, I remark that there are still an old snapshot version of the hexa.binding artifact hanging around. The latest released version is 1.3 and the working version is 1.4-SNAPSHOT so the version 1.3-SNAPSHOTshould not be used anymore. Which is the project still depending on this very deprecated this version ? Let’s ask the question : depends on fr.lteconsulting:hexa.binding:1.3-SNAPSHOT Here it is ! the project rigpa.org:regsys-clients:1.0.0-SNAPSHOT is still using an old snapshot. Let’s arrange that. Pom Explorer is able to change the pom properties and dependencies by itself. Updating this wrong dependency What we want is to change fr.lteconsulting:hexa.binding:1.3-SNAPSHOT tofr.lteconsulting:hexa.binding:1.3 so that the project uses the latest release available. We could desire to change for fr.lteconsulting:hexa.binding:1.4-SNAPSHOT which would be possible with the same command as we’ll see. For that we will use the change gav command : cha ga fr.lteconsulting:hexa.binding:1.3-SNAPSHOT fr.lteconsulting:hexa.binding:1.3 Here is what Pom Explorer answers : So first Pom Explorer finds what needs to be changed in the graph. This might be the project itself and all projects which depend on it. After that the program begins a loop in which all changes are checked and appropriately transformed when needed. For instance changing a dependency version can become changing a property value. Changes are first resolved as described before and they are then transformed in a change list to apply to be applied to pom.xml files. In the ouput, there is first a little warning saying thefr.lteconsulting:hexa.binding:1.3-SNAPSHOT project was not found. That’s normal because the project in now in version 1.4-SNAPSHOT. So there is no need to modify it. Then in the change list section, the changes that are to be applied to pom.xml files are listed. The first one says ‘project not found’ and that’s ok as seen before. The second one says to modify the C:\documents\repos\regsys-clients\pom.xmland change the dependency ([DEPENDENCY])fr.lteconsulting:hexa.binding:1.3-SNAPSHOT tofr.lteconsulting:hexa.binding:1.3. The “causes” message is useful when a change is caused by other changes (as said before a dependency change can become one or several property changes). If we had properties involved, Pom Explorer would have found them and included them in the change set. Now that we reviewed the proposed changes and agreed with them, let’s apply them by using the same command with the -apply flag : cha ga fr.lteconsulting:hexa.binding:1.3-SNAPSHOT fr.lteconsulting:hexa.binding:1.3 -apply We see that at the end of the same process, the program updated the dependency in the right pom.xml file. Let’s have a look at the file it self : fr.lteconsulting hexa.binding 1.3 compile OK, the file is correct now… Oh well no ! I just find other dependencies in SNAPSHOT versions ! Finding more duplicate and obsolete dependencies Let’s accept it, our projects are not up to date. Well let’s see how many of those artifacts there are with multiple versions used. For that i type the checkcommand : Ok there is some work to do ! Opening a version Now let’s look at another use case. Say that the hexa.binding project is in version 1.3 and i want to open the version 1.4-SNAPSHOT. I also want all the projects which depend on version 1.3to move to 1.4-SNAPSHOT. On the way, I want all modified projects still in a release version to be SNAPSHOT-ized too. And i want this to happen recursively as new projects are opened. With Pom Explorer, that’s only one command : change gav fr.lteconsulting:hexa.binding:1.3 fr.lteconsulting:hexa.binding:1.4-SNAPSHOT As you can see, warnings are generated when projects are reopened : Those are normal warnings, they are just here so that you know what happens. Then, there is a big list of changes to be made, because the hexa.bindingartifact is used in many central projects that were in a release state. Glad that we didn’t do that by hand ! Even with the maven-update-version plugin, there would have been a lot of repositories to go to open and update. Let’s apply the changes with the -apply flag : cha ga fr.lteconsulting:hexa.binding:1.3 fr.lteconsulting:hexa.binding:1.4-SNAPSHOT -apply All the changes have been made, about 30 of them. In one go ! Refresh the page so that a new session is created from the changed files. We can see that many of the projects have been reopened : You now have to commit all the repositories with this update. Pom explorer does not do that yet, but maybe in the future ! Releasing many poms Imagine the sprint is almost finished now and it’s now time to release the projects. Type the gav li fr.lteconsulting again to get the GAVs list (fr.lteconsulting is my projects package name, so I filter GAVs with that), choose one and let’s release it : fr.lteconsulting:hexa.binding.samples:1.4-SNAPSHOT The thing in the release is to have all direct and transitive dependencies released too. That’s what Pom Explorer checks. It then generates a change list to materialize your requirements. Other use cases Listing provided and referenced classes You can ask which Java classes are provided and referenced by GAVs. That’s sometimes a useful information to have. Try those commands : classes providedBy fr.lteconsulting:hexa.css:1.3 classes referencedBy fr.lteconsulting:hexa.css:1.3 Optimizing your project’s dependencies Sometimes, you ask yourself “do I still need this and that dependency ?” but you are not very sure, and since you lack time to investigate, eventually the dependency stays in your project for a long time, causing of course maintenance issues sometime. Let’s have Pom Explorer help us in the quest for the obsolete dependency. garbage dep fr.lteconsulting:carousel:1.0-SNAPSHOT This will give you something like that : You can refer to the project documentation to find how to use those informations. But sure that it can help you give away those useless dependencies ! Other goodies : graphs ! Pom Explorer can do two other things to help you visualize your dependency graph : export GraphML files so you can use them in another graph software (like yEd for instance). display an interactive 3d graph Exporting GraphML files GraphML is an open format to describe graphs. With the graph exportcommand, you can get graphml files of your working session. The program will create two files and display the links to them. Those two files are corresponding to two graphs : the dependency graph as usual, and the dependency graph between the git repositories containing your projects. Sometime one git repository can contain multiple projects and a view of the dependencies at the repository level is useful in those cases. This is the kind of picture you can get easily from editors like yEd : Interactive 3D graph Thanks to the WebGL standard which allows direct access to the 3D hardware on the running machine and thanks to libraries like three.js and ngraph.pixel, it is possible to display an interacive 3d graph. More over it is possible to customize the appearance of the graph to give account of different perspectives. Type the graph command and click on the link. This will open another tab containing the living 3d graph of your projects. When focus is given to the 3d viewport, the W, A, S, F and arrow keys allow to move in the 3d space. On the right, there is a text area where you can edit some javascript callback to customize the graph appearance. You can also stop the moving of the particle with the checkbox at the bottom right of the screen. It is not necessarilly useful, but sometimes it is relaxing to admire your work in the form of a living and moving graph ! Conclusion There are many other functions in Pom Explorer, but they are for you to discover now. This tool finds easily its place in the daily workflow because of the functions it provides. The fact that one can run it locally or on a shared server allows to use it as you wish. It is still in early development phase so many more functionalities could come up. On this subject, don’t hesitate submitting a little pull request on the GitHub repository… Pom Explorer is made with love by LTE Consulting
June 26, 2015
by Arnaud Tournier
· 13,214 Views
article thumbnail
.NET Deployment Tips From New Relic Community Forum Users
[This article was written by Wyatt Lindsay] New Relic’s Community Forum is designed to be a place for our users to share their experiences, questions, problems, and fixes. The collective expertise and creativity of the New Relic community has generated some outstanding solutions to everyday issues, and we want to call out some of them in the area of .NET agent deployment excellence. Basic installation Installing New Relic’s .NET agent is designed to be simple: run the installer on the target host and choose the features you want to include. For Microsoft Azuredeployments, install one of our NuGet packages. Installation requires a reset of IIS for the software to load into your application. To upgrade, we recommend first stopping IIS, installing the newer agent version, and then starting up IIS again. It’s also possible to perform a “silent” (manual) install using msiexec.exe, for example: msiexec.exe /i C:\NewRelicAgent.msi /qb NR_LICENSE_KEY= INSTALLLEVEL=1 See the documentation for complete manual installation options. Leveraging PowerShell Scripting the silent installation provides convenience and flexibility. Here’s an example script provided by community member Jon Carl in this post: $msiName = $licenseKey = $arguments = "/i $msiName /L*v install.log /qn NR_LICENSE_KEY=$licenseKey" if ($msiName -ne $null) { $exitCode = (Start-Process -FilePath "msiexec" -ArgumentList $arguments -Wait -PassThru).ExitCode; if($exitCode -eq 0) { Write-Host "Installation successful!" -ForegroundColor Green } else { Write-Host "Installation unsuccessful. Exitcode: $exitCode" -ForegroundColor Red } } This script works great when run directly on the target machine. Another forum user (Kym McGain) noticed that the installation didn’t complete before the session ended when executing the script remotely. This caused the installer to quit partway through. Kym posted this script that uses a ‘while’ loop to ensure the installer completes. As a bonus, it stops IIS before and restarts it after the software installs. As mentioned above, these steps are usually needed when upgrading. $installNewRelic = { $runProcess = { param($process,$arguments) $res = Start-Process -FilePath $process -ArgumentList $arguments -Wait -PassThru while ($res.HasExited -eq $false) { Write-Host "Waiting for $process..." Start-Sleep -s 1 } $exitCode = $res.ExitCode if($exitCode -eq 0) { Write-Host "$process successful!" -ForegroundColor Green } else { Write-Host "$process unsuccessful. Exitcode: $exitCode" -ForegroundColor Red } } $msiName = $licenseKey = $arguments = "/i $msiName /L*v install.log /qn NR_LICENSE_KEY=$licenseKey" Invoke-Command $runProcess -ArgumentList "IISRESET","/STOP" Invoke-Command $runProcess -ArgumentList "msiexec.exe",$arguments Invoke-Command $runProcess -ArgumentList "IISRESET","/START" } Chef, Puppet, and Chocolatey Deployment options abound for modern Web developers. These solutions often require a known download path and installer name. New Relic offers a consistent filepath and MSI name for the agent in an effort to make automated deployment easier for .NET customers: http://download.newrelic.com/dot_net_agent/release/NewRelicDotNetAgent_x64.msi http://download.newrelic.com/dot_net_agent/release/NewRelicDotNetAgent_x86.msi Several Community members have created packages for these utilities. Chocolatey users are invited to use the following NuGet package created by kireevco: https://chocolatey.org/packages/newrelic-dotnet New Relic community member ePitty built a Puppet module to handle .NET agent deployment: https://github.com/epitty1023/puppet-newrelicappmon Chef users, meanwhile, can check out the following cookbook for .NET and many other platforms New Relic supports: http://community.opscode.com/cookbooks/newrelic New Relic Community member E_Bow wrote a Chef recipe that goes a step further by stopping IIS before the installation and starting it again after completion: #Stop IIS iis_site 'Website' do action [:stop] end # install latest Newrelic agent from web include_recipe 'newrelic::repository' include_recipe node['newrelic']['dotnet-agent']['dotnet_recipe'] license = node['newrelic']['application_monitoring']['license'] windows_package 'Install New Relic .NET Agent' do source node['newrelic']['dotnet-agent']['https_download'] options "/qb NR_LICENSE_KEY=#{license} INSTALLLEVEL=#{node['newrelic']['dotnet-agent']['install_level']}" installer_type :msi action :install end #Start IIS iis_site 'Website' do action [:start] end The author states that they were unable to pull the New Relic license key from the configuration JSON in Chef Overrides, requiring them to modify the config file on each machine and manually enter the key. We invite any Chef experts out there to extend and improve this recipe so that it correctly pulls the license key. We are continually impressed by the smarts and spirit of our New Relic Community Forum members, and jump at the chance to highlight their contributions. Look for more Forum projects in the New Relic blog in the future. Do you have your own approach, tips, or recipes? Please share them in the New Relic Community Forum.
June 26, 2015
by Fredric Paul
· 1,658 Views
article thumbnail
DevOps Leadership Series: Compliance, Testing, and Rugged
This past week, I headed to London for the Rugged DevOps event, where I had the chance to catch up with a few more industry thought leaders. First, I caught up with Gareth Rushgrove from Puppet Labs, who also runs the DevOps Weekly newsletter. In this episode of the DevOps Leadership Series, Gareth explains why the importance of security tests within DevOps practices is going to be a big topic for years to come. Gareth anticipates that “In five years time we will look back on this and ask ‘why were we not always doing this’?” I then caught up with Helen Beal, Head of DevOps at Ranger4, where she discussed the importance of DevOps and security. While the two ultimately need to work hand and hand, she voiced concerns about DevOps practices sometimes circumventing controls that are essential to a business’s safety. She also said that DevOps supports security in a number of ways: from making things consistent to relying on more automation. Finally, I caught up with Justin Arbuckle, Vice President, EMEA & Chief Enterprise Architect at Chef. He discusses key trends he expects to see in over the next 9-12 months regarding compliance and security. Justin explains that compliance as a core driver of DevOps is something that will start to shape our conversations over the next year, whereas the two have previously been seen as contradictory. He believes that we will see that high velocity organizations are able to improve compliance continuously, and tells us to look out for the changing role of the security officer in 2015. Next up in the series, I head to the United States Capitol, Washington DC, for DevOps Days DC. NOTE: If you have missed any of the other videos from this series, you can find them here. (We’re up to 15 so far).
June 26, 2015
by Derek Weeks
· 1,643 Views
article thumbnail
From Design to Execution with JBoss BPM Suite & Signavio Process Editor
Occasionally we are asked about JBoss BPM Suite integration with other products and layers in an enterprises architecture. We have published articles talking about how to achieve this with various aspects such as: Microservices integration Data integration Articles are one thing, but seeing is believing, so we have done a few webinars to show you live how to tackle integration: Data integration webinar PEX webinar Along with these articles we have always published demo projects that give you a closer look and chance to get hands on with these integration strategies: JBoss BPM Suite & JBoss Fuse Travel microservices story JBoss BPM Suite & JBoss Data Virtualization integration Imported Signavio Process Editor mortgage workflow. There is another integration story yet to be told about how one can leverage other tooling together with JBoss BPM Suite. This article will introduce one such company,Signavio, that provides a Signavio Process Editor so"...you can start modeling and engaging your organization in improving operational efficiency through the development of optimal models..." The following demo project provides a working example of how you can model an example mortgage process in Signavio Process Editor and then bring it into JBoss BPM Suite where you can add implementation details, integration details and other implementation details to finally execute the mortgage process end-to-end. Demo project As always we bring you not only a story, but a reusable demo project you can easily spin up yourself to explore the details around how a JBoss BPM project would integrate with the model designed in Signavio Process Editor. The project is called the JBoss BPM Suite & Signavio Process Editor Integration Demo. The project installs JBoss BPM Suite 6.1 with an example mortgage project with rules, process, forms and other artifacts. It also includes a copy of an exported Signavio Process Editor mortgage process that we then show how to import. Final mortgage workflow project with implementation details and integration details completed. Ready to run! This gives you the initial starting point after importing the Signavio process and the completely integrated final mortgage project that you can run side-by-side. To setup this project there are just a few simple steps to get going and will be up and running minutes: Installation Download and unzip. Add products to installs directory. Run 'init.sh' or 'init.bat' file. 'init.bat' must be run with Administrative privileges. Start JBoss BPMS Server by running 'standalone.sh' or 'standalone.bat' in the /target/jboss-eap-6.1/bin directory. Login to http://localhost:8080/business-central - login for admin, appraisor, broker, and manager roles (u:erics / p:bpmsuite1!) Mortgage Loan demo pre-installed as project. Using process designer, import the Signavio process that was exported to the file found in: support/MortgageDemoSignavio.bpmn Looking to Automate your business? See screenshots provided in project for how this should look and note that the JBoss BPM Suite process designer included validation that puts messages about tasks not specified, this is correct as at this point you need to start implementing the process tasks. You can examine the imported process and note the various details captured during initial workshops have been put into the process details for each step in the workflow. After implementing these steps you will find the final process ready to run. You can now explore the final project by deploying it and starting a new instance. We hope you enjoy this example project and feel free to browse for more at JBoss Demo Central.
June 26, 2015
by Eric D. Schabell DZone Core CORE
· 1,880 Views · 1 Like
article thumbnail
OpenStack + Private Cloud = Ideal Habitat for Devops
The use of OpenStack in the private cloud is invaluable for DevOps. It provides engineers the ability to innovate quickly and deal with uncertainty. It also maximizes existing infrastructure and provides a programmable, software-defined IaC. Openstack in the private cloud = agile development OpenStack has emerged as the de facto standard for IaaS in the private cloud. It gives engineers a vital self-service capability to provision (and de-provision) environments, allowing them to act autonomously, in the moment. This helps to eliminate the downstream bottleneck caused by waiting for operations staff to find time to do the provisioning. As OpenStack is open source it is vendor agnostic, allowing you to take advantage of competitive pricing rather than suffering from vendor lock-in. A private cloud means lower cost for the same capacity in a public cloud, which is especially useful for enterprises with high data needs. For security reasons, OpenStack is still mainly used in the private cloud by developers and QA, i.e. in a non-production context. However, OpenStack gives an ability to optimize application performance and/or security by having more control compared to public cloud. The software is increasingly backed by the critical mass of leading IT infrastructure vendors such as IBM, CICSO and HP. Gartner assumes that “by 2019, OpenStack enterprise deployments will grow tenfold, up from just hundreds of production deployments today, due to increased maturity and growing ecosystem support.”1 Challenges to consider OpenStack implementation skills are still rare in the market, so experimentation and self-learning is necessary. Although this takes time, it is offset by the fact the software is free and represents a good opportunity to gain internal expertise. This is particularly valid if you class infrastructure as a core competence. The maturity and functionality of OpenStack projects vary widely - while it covers storage, network and compute, the main adoption currently happens around compute (Nova) and block storage (Cinder), with object storage and network (Neutron) lacking significantly behind. However, without leveraging virtualized network services as part of a private cloud, full-stack environment provisioning is not possible, so don’t forget to add necessary network services to your private cloud. Where to begin Integrating OpenStack clouds with existing infrastructure can be a challenge. It is hardly plug and play. At first, it is best to focus on relatively isolated DevOps environments, such as Gartner’s “mode two”2 applications rather than introducing open stack across the board straight away, (Bimodal IT “refers to having two modes of IT, each designed to develop and deliver information – and technology – intensive services in its own way. Mode 1 is traditional, emphasizing scalability, efficiency, safety and accuracy. Mode 2 is nonsequential, emphasizing agility and speed.”3) As with any open source software, new functions and upgrades are frequently released. This means keeping up with changes in functionality and filling gaps with customizations or third-party products. Upgrades are complex and typically require planned downtime. For these reasons, we recommend choosing a hardened distribution and sticking with it. Openstack is the most complete vendor agnostic solution for storage, network and compute services. The ability for developers to instantly spin up environments at any time is invaluable for a fully agile DevOps environment, and is well worth the effort it takes to acclimatize to Openstack. 1 http://www.prnewswire.com/news-releases/suse-openstack-cloud-5-to-simplify-private-cloud-management-300048721.html 2 http://www.gartner.com/it-glossary/bimodal 3 http://www.gartner.com/it-glossary/bimodal
June 26, 2015
by Ron Gidron
· 3,967 Views · 2 Likes
article thumbnail
Simplified API Monitoring for DevOps Teams
[This article was written by Laura Strassman] AlertSite is now integrated with Ready! API. This means that developers, testers and operations teams can collaborate together on API quality using the same tests and metrics, simplifying configuration of monitoring assets and ultimately turning around performance problems in real time. There are several advantages to this approach: You should be monitoring your APIs in production. When your API moves into production from test, the environment changes – there is no way to know if theAPI performanceis compromised unless you look. Furthermore, you can find problems that may be a result of the location or variance in traffic. There is no easier method. You can simply click a button from right in the Ready! API interface and see the status of your APIs in production it can’t get any easier. You take your already created test cases and turn them into monitors whenever you have a new API or test you want to keep an eye on. Troubleshooting is like shooting fish in a barrel. You wrote the functional test, you know it works, and when something comes back as not working you can isolate it quickly. All of this makes it easy to be ahead of problems, solve them quickly when they happen and keep customers happy. You can be monitoring your APIs in less than 3 minutes: <br>
June 25, 2015
by Denis Goodwin
· 1,518 Views
article thumbnail
How to Debug Your Maven Build with Eclipse
When running a Maven build with many plugins (e.g. the jOOQ or Flyway plugins), you may want to have a closer look under the hood to see what’s going on internally in those plugins, or in your extensions of those plugins. This may not appear obvious when you’re running Maven from the command line, e.g. via: C:\Users\jOOQ\workspace>mvn clean install Luckily, it is rather easy to debug Maven. In order to do so, just create the following batch file on Windows: @ECHO OFF IF "%1" == "off" ( SET MAVEN_OPTS= ) ELSE ( SET MAVEN_OPTS=-Xdebug -Xnoagent -Djava.compile=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005 ) Of course, you can do the same also on a MacOS X or Linux box, by usingexport intead of SET. Now, run the above batch file and proceed again with building: C:\Users\jOOQ\workspace>mvn_debug C:\Users\jOOQ\workspace>mvn clean install Listening for transport dt_socket at address: 5005 Your Maven build will now wait for a debugger client to connect to your JVM on port 5005 (change to any other suitable port). We’ll do that now with Eclipse. Just add a new Remote Java Application that connects on a socket, and hit “Debug”: That’s it. We can now set breakpoints and debug through our Maven process like through any other similar kind of server process. Of course, things work exactly the same way with IntelliJ or NetBeans. Once you’re done debugging your Maven process, simply call the batch again with parameter off: C:\Users\jOOQ\workspace>mvn_debug off C:\Users\jOOQ\workspace>mvn clean install And your Maven builds will no longer be debugged. Happy debugging!
June 25, 2015
by Lukas Eder
· 25,037 Views
article thumbnail
Spring Integration Kafka 1.2 is Available, With 0.8.2 Support and Performance Enhancements
Spring Integration Kafka 1.2 is out with a major performance overhaul.
June 25, 2015
by Pieter Humphrey
· 2,960 Views
article thumbnail
Investing in Your Infrastructure
Having mentored and invested in startups I have come to learn what works and doesn’t websites. The reality of getting out of startup mode and scaling takes vision and the ability to anticipate how you may need to pivot. There are two main reasons why startups do not scale. The first is a lack of experience and mentorship. This is closely followed by a lack of a working capital. An effective entrepreneur understands not only how to pivot but how to utilize capital investment. This article aims to demystify the need for investing in site infrastructure. When scaling a business there are several huge issues that executives tend to forget. whether you are a startup or a fortune 500 company your website is the organizations public face. Maintaining a clean and secure site will help to avoid deep routed problems that could potentially destroy not only you site by your reputation. Hackers can be devastating and end up costing you millions. Another aspect of your site to consider is funnel optimization. By making this a priority you will be able to effective guide customers step by step into a conversion. American With Disabilities Act (ADA) The ADA can be a freighting legal area for many entrepreneurs. Many think that ADA only applies to physical boundaries such as implementing ramps for wheelchairs and accessible bathroom stalls. However this is far from the case. Few people know that the digital world also counts. Is your website written in HTML5? If not you are in violation of ADA. This newest version of the HTML coding language allows for an audible version of a web site for those facing impaired vision. However there are some quick an easy ways to gain some ADA points to allow for more leeway in other more long-term solutions. Implementing closed captioning into your promotional videos will allow for increased accessibility for those facing an auditory disability. Investing in infrastructure is not a single step but a constant to building a good company culture where employees can feel proud of where they work and remain safe. Please comment on this article if you have any additional suggestions.
June 25, 2015
by David Schwartz
· 1,037 Views
article thumbnail
7 Things I Didn’t Expect to Hear at Gartner’s IT Ops Summit
Last week’s Gartner IT Operations Strategies & Solutions Summit in Orlando, Fla., was exactly what you’d expect—a place to talk about the IT operations issues impacting some of the largest companies in the world. Even so, there were a few interesting surprises. Among them: 1. Bi-modal is big. Not everyone will succeed. Gartner continued to tell its customers to employ two modes of IT—a traditional, slower moving capability for older, typically internal systems of record; and a high-speed, experimental one for new, typically customer-facing Web and mobile apps. “This is a time of experimentation and innovation,” said Gartner VP and distinguished analyst Chris Howard in his opening keynote. Organizations can’t ignore that there are multiple speeds and they should participate in all. Gartner managing VPRonni Colville added that by 2017, 75% of IT orgs will have this “bi-modal” IT capability. See also: Bi-Modal IT: Gartner Endorses Both Disruptive and Conservative Approaches to Technology However, “50% will make a mess of it,” Colville said. Why? Not necessarily because of technology failings, but more often because of a lack of people skills. 2. IT success is all about people. Donna Scott, also a Gartner VP and distinguished analyst, told her keynote audience that “you will be judged on agility, speed, and innovation.” However, the biggest problems Gartner sees for infrastructure and operations team engagement and innovation are lack of time, company culture that’s not conducive to these approaches, and a lack of business skills in IT. More than half of the people responding to an in-room poll said “people” are the part of IT ops that must change first. Not technology. Gartner research director George Spafford underscored similar issues in large organizations trying to use DevOps at scale: people and “human factors” are the biggest concerns from his in-room poll. All these probably contributed to hiring best-selling author Daniel Pink as a keynote speaker on the opening day of the conference. His focus? Not IT or architecture. Instead, he pounded home the importance of influencing people and selling internally. 3. Big orgs are trying DevOps. But the issues are different at scale. In numerous sessions I saw many hands go up when analysts asked, “Who here is trying DevOps?” Clearly, the approach is getting traction in large companies. But there’s lots of learning still to do. In fact, that was Spafford’s biggest bit of advice. “Always be learning,” he said, “trying to see what works and what breaks, especially at scale.” And, even once you’ve had some initial success, keep learning. “If you’ve done ‪DevOps, stay humble,” he advised. 4. Looking to innovative organizations for ideas … analytics on the rise. Many sessions addressed how large organizations are taking on ideas fostered by smaller, more risk-tolerant companies, and offered advice for doing so successfully. In addition to multiple discussions of DevOps, an entire session was devoted to establishing your own “Genius Bar®—a “walk-up IT support center” as explained in this CIO article. As at previous conferences, Gartner research VP Cameron Haight ran several sessions on lessons learned from firms running massive, Web-scale IT systems. “You need lots of data … and access to it inexpensively,” he said. Some commercial monitoring companies (New Relic included!) got a shout out for taking the lessons of Web scale IT to heart in their offerings. In addition, Haight said, “Analytics are increasingly important for application performance monitoring given the huge amount of data now available.” 5. Cloud: Enterprises want it, but aren’t very good at it yet. Gartner research director Dennis Smith talked through the enterprise’s interest in cloud computing. A huge majority of his in-room poll wanted some mix of both public and private cloud, while only 9% wanted to use only a private cloud environment and a measly 4% were looking to move entirely to the public cloud. The most popular choice (41%) was an 80/20 split between private and public cloud infrastructure. “Enterprises don’t make the dean’s list,” for cloud usage, Smith said, earning no more than a C average in his opinion. Large organizations are doing well at visibility, governance, and delivering standardized stacks, he said, but are less skilled at optimizing for these new environments. Still, Smith said the trends point toward enterprises improving on all fronts. 6. Cloud security can be better than yours. Importantly, Gartner VP and distinguished analyst Neil MacDonald gave the cloud a vote of confidence: noting that, for a variety of reasons, “Well-managed public cloud can be more secure than your own data center.” For example, on-premise software can pose serious security risks, he said, because of “deployment lag” where customers are stuck using software releases with unpatched security vulnerabilities. With a cloud-based Software-as-a-Service (SaaS), security updates can be more quickly rolled out to all customers. But cloud security can be different, requiring a shift to information-level security from OS-level security. Best practices include doing away with a huge pool of all-powerful sysadmins in favor of JEA, or “just enough administration,” where sysadmins have just enough privileges to do their job, and no more. An analogous security practice for compute resources is “least privilege,” where apps and microservices can’t talk to each other unless they specifically need to do so. Audience polling supported MacDonald’s optimistic view of cloud security, which suggests that large enterprises may struggle less with their cloud policies moving forward. 7. Containers: Try ’em! Ahead of this week’s DockerCon in San Francisco, Gartner devoted significant airtime to educating the audience on containers and microservices. My summary of ‪Gartner VP and distinguished analyst Tom Bittman’s advice on containers was simple: Try ’em. Now. Complement them with VMs. ‪And Docker (the company) is important, but not the be-all and end-all in this space. Bittman (copping to some deja vu from Gartner presentations he made on server virtualization 13 years ago) noted that while virtualization has been focused on admin and ops functions, containers are focused on value for developers. But because containers are well suited for driving up VM utilization for workloads that share the same OS, we can expect to see more combinations of containers and server virtualization. Finally, Bittman underscored that Gartner doesn’t see containers having much impact on premise, but making a huge difference in the cloud. That doesn’t necessarily fit with what’s been shown in other research, such as this 2015 State of Containers Survey sponsored by VMblog.com and StackEngine, so we’ll want to watch how this plays out. This is all a lot to digest. The Gartner IT Operations Strategies & Solutions Summitacknowledges the importance of dealing with existing IT systems and practices as well as promising new technologies and thinking, and tries to point a way forward. In fact, Haight had a very good quote about microservices that I thought also served to wrap up the entire event: “If you want to run with the big dogs, you need to rethink application architecture,” he said. That can be very difficult for an enterprise to fully implement … but also very appealing. Note: Al Sargent contributed to this post. All product and company names herein may be trademarks of their registered owners. Server, tortoise and hare, business team, and cloud security images courtesy ofShutterstock.com.
June 24, 2015
by Fredric Paul
· 1,820 Views
article thumbnail
Literate Programming and GitHub
I remain captivated by the ideals of Literate Programming. My fork of PyLit (https://github.com/slott56/PyLit-3) coupled with Sphinx seems to handle LP programming in a very elegant way. It works like this. Write RST files describing the problem and the solution. This includes the actual implementation code. And everything else that's relevant. Run PyLit3 to build final Python code from the RST documentation. This should include the setup.py so that it can be installed properly. Run Sphinx to build pretty HTML pages (and LaTeX) from the RST documentation. I often include the unit tests along with the sphinx build so that I'm sure that things are working. The challenge is final presentation of the whole package. The HTML can be easy to publish, but it can't (trivially) be used to recover the code. We have to upload two separate and distinct things. (We could use BeautifulSoup to recover RST from HTML and then PyLit to rebuild the code. But that sounds crazy.) The RST is easy to publish, but hard to read and it requires a pass with PyLit to emit the code and then another pass with Sphinx to produce the HTML. A single upload doesn't work well. If we publish only the Python code we've defeated the point of literate programming. Even if we focus on the Python, we need to do a separate upload of HTML to providing the supporting documentation. After working with this for a while, I've found that it's simplest to have one source and several targets. I use RST ⇒ (.py, .html, .tex). This encourages me to write documentation first. I often fail, and have blocks of code with tiny summaries and non-existent explanations. PyLit allows one to use .py ⇒ .rst ⇒ .html, .tex. I've messed with this a bit and don't like it as much. Code first leaves the documentation as a kind of afterthought. How can we publish simply and cleanly: without separate uploads? Enter GitHub and gh-pages. See the "sphinxdoc-test" project for an example. Also thishttps://github.com/daler/sphinxdoc-test. The bulk of this is useful advice on a way to create the gh-pages branch from your RST source via Sphinx and some GitHub commands. Following this line of thinking, we almost have the case for three branches in a LP project. The "master" branch with the RST source. And nothing more. The "code" branch with the generated Python code created by PyLit. The "gh-pages" branch with the generated HTML created by Sphinx. I think I like this. We need three top-level directories. One has RST source. A build script would run PyLit to populate the (separate) directory for the code branch. The build script would also run Sphinx to populate a third top-level directory for the gh-pages branch. The downside of this shows up when you need to create a branch for a separate effort. You have a "some-major-change" branch to master. Where's the code? Where's the doco? You don't want to commit either of those derived work products until you merge the "some-major-change" back into master. GitHub Literate Programming There are many LP projects on GitHub. There are perhaps a dozen which focus on publishing with the Github-flavored Markdown as the source language. Because Markdown is about as easy to parse as RST, the tooling is simple. Because Markdown lacks semantic richness, I'm not switching. I've found that semantically rich markup is essential. This is a key feature of RST. It's carried forward by Sphinx to create very sophisticated markup. Think:code:`sample` vs. :py:func:`sample` vs. :py:mod:`sample` vs.:py:exc:`sample`. The final typesetting may be similar, but they are clearly semantically distinct and create separate index entries. A focus on Markdown seems to be a limitation. It's encouraging to see folks experiment with literate programming using Markdown and GitHub. Perhaps other folks will look at more sophisticated markup languages like RST. Previous Exercises See https://sourceforge.net/projects/stingrayreader/ for a seriously large literate programming effort. The HTML is also hosted at SourceForge: http://stingrayreader.sourceforge.net/index.html. This project is awkward because -- well -- I have to do a separate FTP upload of the finished pages after a change. It's done with a script, not a simple "git push." SourceForge has a GitHub repository. https://sourceforge.net/p/stingrayreader/code/ci/master/tree/. But. SourceForge doesn't use GitHub.com's UI, so it's not clear if it supports the gh-pages feature. I assume it doesn't, but, maybe it does. (I can't even login to SourceForge with Safari... I should really stop using SourceForge and switch to GitHub.) See https://github.com/slott56/HamCalc-2.1 for another complex, LP effort. This predates my dim understanding of the gh-pages branch, so it's got HTML (in doc/build/html), but it doesn't show it elegantly. I'm still not sure this three-branch Literate Programming approach is sensible. My first step should probably be to rearrange the PyLit3 project into this three-branch structure.
June 24, 2015
by Steven Lott
· 1,963 Views
article thumbnail
Perforce and Go2Group Integrate Helix SCM Platform with ConnectALL ALM Router
New Integration Provides Seamless Connections Between Perforce Helix and Leading Application Lifecycle Management Systems WOKINGHAM, UK. (June 24, 2015) – Perforce Software, the leader in software configuration management (SCM) and collaboration, and Go2Group, an Atlassian Platinum and Enterprise Expert, today announced the Perforce ConnectALL Adapter. The new adapter for Go2Group’s ConnectALL ALM Router connects Perforce Helix to Application Lifecycle Management (ALM) systems supported by ConnectALL. The companies also announced that they have expanded their partnership, which first began in 2002. “Very few SCMs can handle binary data, and no other SCM solution supports large file formats that scale across globally distributed enterprises like Helix,” said Brett Taylor, president of Go2Group. “Our customers demand future-proof solutions, and with Perforce we know they don’t have to worry about outgrowing their systems—it will serve them well whether they’re a team of 50 or 50,000.” With the Perforce adapter, ConnectALL automatically synchronises data and workflow with other ALM systems and integrates ALM systems components within minutes. “We’re excited to be a part of the ConnectALL ecosystem of adapters and to enable companies to more easily design, configure, synchronise, manage, and monitor their integrations with Perforce,” said Dave Robertson, vice president of Channels at Perforce. “We’re glad to extend our partnership with Go2Group to new technologies and markets.” Go2Group is part of Perforce’s network of sales partners across Europe, the Middle East, Africa, Asia Pacific and India. Perforce partners serve customers in more than 100 countries worldwide. The Perforce ConnectALL Adapter is available for purchase from the Go2Group website.
June 24, 2015
by Fran Cator
· 950 Views
article thumbnail
New Integrated Biometrics System Extends Enterprise Access Control to the Data Centre and Other High-Security Settings
Combining BioConnect with Digitus server cabinet access control makes biometric identity management more effective and easier ENTERTECH SYSTEMSandDigitus Biometricshave just announced a new technology partnership between ENTERTECH’S BioConnect identity management platform and the Digitus db Bus and db Cabinet Sentry for server cabinet access control. The result is a new, fully-integrated solution called db BioConnect. The newdb BioConnectlets company data centres, co-location data centres and IT room customers simplify biometric implementation, enrollment and management for access control of perimeter doors, interior rooms, cages and now server racks all integrated into one identity platform. This game-changing solution now extends enterprise access security platforms into the data centre, making investments in Access Control Management Systems far more economical and effective. “Digitus Biometrics is the leader in providing biometric access control to the critical infrastructure market," said Rob Douglas, ENTERTECH SYSTEMS CEO. "With Digitus technology integrated to BioConnect, all 15 of our certified access control partners will now be able to offer it to their customers. For the first time, end users will have access to an integrated biometric solution that secures access control from the data centre entrance to the server cabinet instead of having to deal with stand-alone systems.” The list of BioConnect certified access control partners can be found atwww.bioconnect.com/partners. ENTERTECH SYSTEMS’BioConnectis the most advanced identity management platform on the global market today. Simple, secure and scalable, it provides Suprema biometric authentication across leading access control systems. As an application for security professionals, BioConnect helps enterprises successfully implement identity solutions by making deployment and use of biometrics easier than ever. BioConnect addresses deployment challenges by reducing costs, overcoming complexity and making it easier to on-board users. The platform provides seamless synchronization of data such as new cardholders, changes and deletions, and is tailor-made for enterprises where true identity is critical for secure access to physical facilities and software applications. “Our biometric access control solutions are designed to meet the needs of a diverse range of clients," said David Orischak, Digitus Biometrics CEO. "This technology integration to create db BioConnect will let us offer a single, centralised, highly advanced access control solution that's easy to deploy and use. An industry first, our customers will be able to use one centralised system facility-wide to secure a company’s critical infrastructure and data." db Bus ServerRack Access Controlis designed for data centres needing to secure large volumes of server cabinets with only one small component per cabinet. A single db Bus controller allows the user to secure up to 32 racks with a single 48V power supply and may be infinitely scaled.db Cabinet Sentryis designed for data centres with a structured cabling scheme. It is Digitus’ most compact, cost-effective and energy-efficient means of securing server cabinets. This extremely versatile unit can be deployed in networked or stand-alone environments, using power over Ethernet (PoE) or an external power supply.db BioLockis a server cabinet lock that uses biometric identification with prints for up to 10 fingers per user. Through this new technology partnership, these db products will be integrated with BioConnect to create the new integrated db BioConnect solution. “Digitus’ use of leading Suprema biometric technology in their server cabinet access control solutions is a natural fit for ENTERTECH SYSTEMS as the operating partner for Suprema in the US, Canada, UK, Ireland and Puerto Rico,” added Douglas. “The implications to the market are significant," added Orischak. "The integrated db BioConnect solution can be used to manage any cabinet system where biometric access control is warranted – even SCIF’s and locked areas housing sensitive assets such as pharmaceuticals, hazardous materials, intelligence archives, customer and patient records, as well as critical IP.” For more information on the db BioConnect integrated solution, visitwww.bioconnect.com/db.
June 24, 2015
by Fran Cator
· 1,338 Views
article thumbnail
How to Split Single PST File into Multiple PSTs & Merge Multiple PSTs using .NET
This technical tip explains how to .NET developers can split and Merge PST files using Aspose.Email. Aspose.Email API provides the capability to split a single PST file into multiple PST files of desired file size. It can also merge multiple PST files into a single PST file. Both the splitting and merging of PSTs operations can be tracked by adding events to these operations. Aspose.Email for .NET is a set of components allowing developers to easily implement email functionality within their ASP.NET web applications, web services & Windows applications. It Supports Outlook PST, EML, MSG & MHT formats. //Code sample for Splitting a Single PST into multiple PSTs [C#] using (PersonalStorage pst = PersonalStorage.FromFile(@"D:\test\source.pst")) { // The events subscription is an optional step for the tracking process only. pst.StorageProcessed += PstSplit_OnStorageProcessed; pst.ItemMoved += PstSplit_OnItemMoved; // Splits into pst chunks with the size of 5mb pst.SplitInto(5000000, @"D:\test\chunks\"); } [VB.NET] Using pst As PersonalStorage = PersonalStorage.FromFile("D:\test\source.pst") ' The events subscription is an optional step for the tracking process only. pst.StorageProcessed += PstSplit_OnStorageProcessed pst.ItemMoved += PstSplit_OnItemMoved ' Splits into pst chunks with the size of 5mb pst.SplitInto(5000000, "D:\test\chunks\") //Code sample for Merging of Multiple PSTs into a single PST [C#] totalAdded = 0; using (PersonalStorage pst = PersonalStorage.FromFile(@"D:\test\destination.pst")) { // The events subscription is an optional step for the tracking process only. pst.StorageProcessed += PstMerge_OnStorageProcessed; pst.ItemMoved += PstMerge_OnItemMoved; // Merges with the pst files that are located in separate folder. pst.MergeWith(Directory.GetFiles(@"D:\test\sources\")); Console.WriteLine("Total messages added: {0}", totalAdded); } [VB.NET] totalAdded = 0 Using pst As PersonalStorage = PersonalStorage.FromFile("D:\test\destination.pst") ' The events subscription is an optional step for the tracking process only. pst.StorageProcessed += PstMerge_OnStorageProcessed pst.ItemMoved += PstMerge_OnItemMoved ' Merges with the pst files that are located in separate folder. pst.MergeWith(Directory.GetFiles("D:\test\sources\")) Console.WriteLine("Total messages added: {0}", totalAdded) End Using //Code sample Merging Folders from another PST [C#] totalAdded = 0; using (PersonalStorage destinationPst = PersonalStorage.FromFile(@"D:\test\destination.pst")) using (PersonalStorage sourcePst = PersonalStorage.FromFile(@"D:\test\source.pst")) { FolderInfo destinationFolder = destinationPst.RootFolder.AddSubFolder("FolderFromAnotherPst"); FolderInfo sourceFolder = sourcePst.GetPredefinedFolder(StandardIpmFolder.DeletedItems); // The events subscription is an optional step for the tracking process only. destinationFolder.ItemMoved += destinationFolder_ItemMoved; // Merges with the folder from another pst. destinationFolder.MergeWith(sourceFolder); Console.WriteLine("Total messages added: {0}", totalAdded); } [VB.NET] totalAdded = 0 Using destinationPst As PersonalStorage = PersonalStorage.FromFile("D:\test\destination.pst") Using sourcePst As PersonalStorage = PersonalStorage.FromFile("D:\test\source.pst") Dim destinationFolder As FolderInfo = destinationPst.RootFolder.AddSubFolder("FolderFromAnotherPst")t Dim sourceFolder As FolderInfo = sourcePst.GetPredefinedFolder(StandardIpmFolder.DeletedItems) ' The events subscription is an optional step for the tracking process only. destinationFolder.ItemMoved += destinationFolder_ItemMoved ' Merges with the folder from another pst. destinationFolder.MergeWith(sourceFolder) Console.WriteLine("Total messages added: {0}", totalAdded) End Using End Using //Helping Methods void destinationFolder_ItemMoved(object sender, ItemMovedEventArgs e) { totalAdded++; } void PstMerge_OnStorageProcessed(object sender, StorageProcessedEventArgs e) { Console.WriteLine("*** The storage is merging: {0}", e.FileName); } void PstMerge_OnItemMoved(object sender, ItemMovedEventArgs e) { if (currentFolder == null) { currentFolder = e.DestinationFolder.RetrieveFullPath(); } string folderPath = e.DestinationFolder.RetrieveFullPath(); if (currentFolder != folderPath) { Console.WriteLine(" Added {0} messages to \"{1}\"", messageCount, currentFolder); messageCount = 0; currentFolder = folderPath; } messageCount++; totalAdded++; } void PstSplit_OnStorageProcessed(object sender, StorageProcessedEventArgs e) { if (currentFolder != null) { Console.WriteLine(" Added {0} messages to \"{1}\"", messageCount, currentFolder); } messageCount = 0; currentFolder = null; Console.WriteLine("*** The chunk is processed: {0}", e.FileName); } void PstSplit_OnItemMoved(object sender, ItemMovedEventArgs e) { if (currentFolder == null) { currentFolder = e.DestinationFolder.RetrieveFullPath(); } string folderPath = e.DestinationFolder.RetrieveFullPath(); if (currentFolder != folderPath) { Console.WriteLine(" Added {0} messages to \"{1}\"", messageCount, currentFolder); messageCount = 0; currentFolder = folderPath; } messageCount++; }
June 24, 2015
by David Zondray
· 1,594 Views
article thumbnail
Writing a Download Server Part I: Always Stream, Never Keep Fully in Memory
Downloading various files (either text or binary) is a bread and butter of every enterprise application. PDF documents, attachments, media, executables, CSV, very large files, etc. Almost every application, sooner or later, will have to provide some form of download. Downloading is implemented in terms of HTTP, so it's important to fully embrace this protocol and take full advantage of it. Especially in Internet facing applications features like caching or user experience are worth considering. This series of articles provides a list of aspects that you might want to consider when implementing all sorts of download servers. Note that I avoid "best practices" term, these are just guidelines that I find useful but are not necessarily always applicable. One of the biggest scalability issues is loading whole file into memory before streaming it. Loading full file into byte[] to later return it e.g. from Spring MVC controller is unpredictable and doesn't scale. The amount of memory your server will consume depends linearly on number of concurrent connections times average file size - factors you don't really want to depend on so much. It's extremely easy to stream contents of a file directly from your server to the client byte-by-byte (with buffering), there are actually many techniques to achieve that. The easiest one is to copy bytes manually: @RequestMapping(method = GET) public void download(OutputStream output) throws IOException { try(final InputStream myFile = openFile()) { IOUtils.copy(myFile, output); } } Your InputStream doesn't even have to be buffered, IOUtils.copy() will take care of that. However this implementation is rather low-level and hard to unit test. Instead I suggest returning Resource: @RestController @RequestMapping("/download") public class DownloadController { private final FileStorage storage; @Autowired public DownloadController(FileStorage storage) { this.storage = storage; } @RequestMapping(method = GET, value = "/{uuid}") public Resource download(@PathVariable UUID uuid) { return storage .findFile(uuid) .map(this::prepareResponse) .orElseGet(this::notFound); } private Resource prepareResponse(FilePointer filePointer) { final InputStream inputStream = filePointer.open(); return new InputStreamResource(inputStream); } private Resource notFound() { throw new NotFoundException(); } } @ResponseStatus(value= HttpStatus.NOT_FOUND) public class NotFoundException extends RuntimeException { } Two abstractions were created to decouple Spring controller from file storage mechanism.FilePointer is a file descriptor, irrespective to where that file was taken. Currently we use one method from it: public interface FilePointer { InputStream open(); //more to come } open() allows reading the actual file, no matter where it comes from (file system, database BLOB, Amazon S3, etc.) We will gradually extend FilePointer to support more advanced features, like file size and MIME type. The process of finding and creatingFilePointers is governed by FileStorage abstraction: public interface FileStorage { Optional findFile(UUID uuid); } Streaming allows us to handle hundreds of concurrent requests without significant impact on memory and GC (only a small buffer is allocated in IOUtils). BTW I am using UUID to identify files rather than names or other form of sequence number. This makes it harder to guess individual resource names, thus more secure (obscure). More on that in next articles. Having this basic setup we can reliably serve lots of concurrent connections with minimal impact on memory. Remember that many components in Spring framework and other libraries (e.g. servlet filters) may buffer full response before returning it. Therefore it's really important to have an integration test trying to download huge file (in tens of GiB) and making sure the application doesn't crash. Writing a download server Part I: Always stream, never keep fully in memory Part II: headers: Last-Modified, ETag and If-None-Match Part III: headers: Content-length and Range Part IV: Implement HEAD operation (efficiently) Part V: Throttle download speed Part VI: Describe what you send (Content-type, et.al.) The sample application developed throughout these articles is available on GitHub.
June 24, 2015
by Tomasz Nurkiewicz
· 17,127 Views
article thumbnail
Overcoming Barriers to Performance and Scalability Test Automation
[This article was written by Ophir Prusak] Guest author Ophir Prusak is chief evangelist atBlazeMeter. To learn more about load and performance testing automation, he invites readers toattend a meetupthis Wednesday, June 24, at New Relic’s San Francisco offices. Performance and load testing are kind of like flossing your teeth. You know you need to do it, but you might not be doing it as much as you should. When your site goes down because it couldn’t handle the load, you look back and realize you might have easily prevented it with a little more testing in advance. That’s why companies are automating their application testing in an effort to lower costs, increase efficiency, and reduce the time needed to release new features. The importance of automated testing in a continuous delivery era Continuous Delivery (CD) is rapidly emerging as the “new normal” in software development, as Perforce discovered in an independent survey, with an estimated 80% of SaaS companies and 51% of non-SaaS companies adopting this practice. Companies that provide Software-as-a-Service know they need to be continuously creating new features, updating their websites, and optimizing their backend. But while software development has adapted nicely in terms of automation, the testing side has moved more slowly. For a fully Continuous Delivery and Integration process to be realized, performance testing must be automated. As the need for testing increases, doing it manually can dramatically increase your time to release. Automating testing throughout the CD process can help detect errors instantly and deliver software faster. Making it work JMeter is the de facto standard in open source load testing. It’s the most widely used open source tool for performance testing for a good reason. There’s virtually nothing it can’t test (websites, native mobile applications, APIs, and Web applications) and it’s extremely powerful and fully featured. Yet there are challenges. JMeter poses a steep learning curve in terms of integration and ease of use. Additionally, it doesn’t integrate easily with APM and Continuous Integration (CI) tools. Many developers have been looking for a way to conduct performance testing with less time and effort—and fewer hiccups along the way. Taurus: An effort to simplify test automation A new open source project called Taurus (Test AUtomation Running Smoothly) is designed to provide exactly that—a way to remove most of the pain of using JMeter on its own. Taurus can give you the ability to Create and define a load test even without using JMeter. Override existing JMeter files or tests configurations. Create human-readable configuration files and testing scripts that are easily added to source control systems like GitHub. Integrate into CI tools like Jenkins. Run multiple tests in parallel. Provide pass/fail criteria back into the CI tool for easier automation of test-results analysis. Make analysis of test results easier and more intuitive. Taurus still uses JMeter under the hood, but is designed to have a much easier learning curve, especially for simple tests. Taurus also offers a built-in result analysis engine that provides both console-based reporting features and result analysis. Performance testing and optimizing your applications is not simple, yet there are solutions available that make the process easier and more successful. I’m looking forward to seeing how the technology evolves even further in the near future. If you want to learn more about Taurus, check out the project on GitHub. Better yet, you are invited to come to a meetup this Wednesday, June 24, at New Relic’s San Francisco offices. You can learn a lot more about Taurus and how you can use it to help scale load and performance testing automation.
June 24, 2015
by Fredric Paul
· 1,762 Views
article thumbnail
Don't Leave Your Database Behind
Microsoft’s recent announcement that Windows 10 will be the last big-bang release for the operating system was a surprise to many. But what on Earth is going on? Why have they suddenly turned round and decided to upgrade their software with a series of frequent updates instead? The answer is a term not many chief executives will be familiar with, but an important one nevertheless: continuous delivery. Continuous delivery for software applications means that new features can be released faster and companies can be more competitive. Although it requires a change in processes and an investment in the tools that make it possible, the return on investment is high – and proven. The 2014 State of DevOps Report from Puppet Labs found that high-performing IT organisations that use practices such as continuous delivery are twice as likely to exceed their profitability, market share and productivity goals. WHAT ABOUT THE DATABASE? A stumbling block many organisations find in their pursuit of continuous delivery is the database. Databases hold critical information and their stability is vital to the bottom line. Often they’re tied into those same applications that can gain from continuous delivery, but they’re seen as too risky to include in the process. In fact, the DZone Guide to Continuous Delivery 2015 showed that while 61 per cent of companies have already implemented continuous delivery for their applications, it falls to half that number when it comes to the database.The result? Application development is faster, smoother and more cost effective, while database development lags behind. “Clients such as Yorkshire Water are seeing a return on investment of 700 per cent after investing in continuous delivery for their databases” At Redgate, we’ve been working on a way to resolve the issue with a Database Lifecycle Management (DLM) solution: one that brings all the advantages of continuous delivery to the database, while protecting the data at the same time. We have some pedigree here. Our tools are already used in more than 90 per cent of Fortune 100 companies, and are trusted in areas such as finance, healthcare and manufacturing, where performance and reliability are not optional. It’s working too. Clients such as Yorkshire Water are seeing a return on investment of 700 per cent after investing in continuous delivery for their databases. Similarly, StateServ, a medical equipment supplier with customers across the United States, has adopted Redgate’s SQL Source Control and SQL Compare to reduce the time it takes to release new versions of its website by 50 per cent. Property services provider Move With Us has also adopted these tools to reduce the cost of releasing database updates by 97 per cent. As Anthony Hall, IT operations manager, says: “We spend less time deploying and more time developing better software, which keeps us ahead of our competitors. It results in a product that more closely matches what the customer wants, at a cheaper price. That equals happy customers, a happier development team and bigger profits for the company.” I love comments like that. It demonstrates that all the effort we’ve put into developing our tools and the DLM process to go with it is paying off. Not for us – for our clients. By aligning the continuous delivery of the application with the continuous delivery of the database, a company will inevitably see increased productivity, reduced risk and higher quality end-products. This suddenly means that your technology becomes a strategic advantage, as opposed to an unpredictable risk, and can be used to deliver real financial benefits to the business. CASE STUDY KEEPING DATA FLOWING FOR YORKSHIRE WATER Yorkshire Water manage the collection, treatment and distribution of water in Yorkshire, supplying around 1.24 billion litres of drinking water daily. At the same time they collect, treat and dispose of about one billion litres of waste water safely back into the environment. As might be imagined, their applications and databases are diverse and technically challenging, and deploying changes to the databases used to be manual and time consuming. The company turned to Redgate’s database development tools to automate the changes, saving time as well as avoiding errors. Using SQL Source Control and SQL Compare, they achieved every aim. In 25 days, they moved their first project from a manual deployment process to full automation and it now takes half a day to start auto-deploying a new project. As software development team leader Carl Davison says: “It’s now a minor overhead to deploy. We predict the return on investment to be in the order of 700 per cent over the next five years. We can deploy database changes as soon as the business needs them, without delays or problems.”
June 24, 2015
by Moe Long
· 1,505 Views
article thumbnail
Hazelcast Cluster Quorum
Originally written by David Brimley. The Death Spiral. A new feature in the 3.5 release of Hazelcast is the Cluster Quorum. In this instance we’re not talking about a Quorum in its traditional distributed systems sense, think of a Cluster Quorum as a kind of gatekeeper, protecting your cluster during times of unexpected member loss. You can use Cluster Quorums to restrict operations on Maps or indeed the entire cluster based upon environmental criteria. This sounds great you say, but I’m still not sure how this can help me? OK. Let's take a look at a scenario… Imagine a cluster that has a very high number of writes to a certain map. We also have other maps that are not updated quite so frequently and all the while we have hundreds of clients all reading from the cluster but not at the same frequency as the data that is entering the system. In normal circumstances if a machine or a number of machines were to die in the cluster we may still have enough memory available to store our data, but the amount of threads available to process requests would be reduced. We now have less cores available and the partition threads in the cluster could quickly become overwhelmed by the one map that is updated rapidly. This could mean other clients becoming starved of threads, unable to service requests. It’s also possible that the remaining members would become so consumed that they’re unable to respond to membership pings, the knock on effect could result in the member being forced out of the cluster on the assumption that it is dead. To protect the rest of the cluster in the event of member loss we need a way to stop the writes to the high frequency map whilst allowing operations to the other data structures. We can then continue to provide a good service to our other users whilst the crashed machines are restored to the cluster. Bring on the Quorum! As of Hazelcast 3.5 we now have the ability to restrict operations on distinct data structures. We do this via a Quorum configuration. We observed that other IMDG products provide Quorums that have protection at a cluster level,we decided to go one step further and provide Quorum protection around data structures as well. In the example below we create a very simple Quorum on the default map. The ‘default’ map in Hazelcast is the configuration used if no other match is found. In this instance no operations will be allowed unless the cluster has a minimum of 3 members. You’ll also note that the Quorum configuration is separate from the Map. This means that you can have multiple Quorums in a cluster attached to many different structures. If the Quorum thresholds are not satisfied then a QuorumException is thrown when we try to interact with the default map in any way. Be it from a client or another member. 3 quorumRuleWithThreeNodes Quorum Functions It’s simple to set up a Quorum check based on cluster size as we’ve seen above, but if you want to make a slightly more complex check you can do this by applying a Quorum Function. QuorumConfig quorumConfig = new QuorumConfig(); quorumConfig.setName("MyQuorum"); quorumConfig.setEnabled(true); quorumConfig.setType(QuorumType.WRITE); quorumConfig.setQuorumFunctionImplementation(new QuorumFunction() { @Override public boolean apply(Collection members) { return (members.size() >= 3) && (someOtherExternalClusterState); } }); In the example above we use Configuration API to set-up the Quorum to disallowwrites if the boolean returned from the QuorumFunction is false. In the function we test if the size of the cluster is greater than 3 and also if a variable namedsomeOtherExternalClusterState is equal true. You now get the idea that by using a function you can test for other state and not just cluster member. Listen In. Another nice feature of Quorums is the ability to listen in to Quorum Events. You can register a new callback interface called not surprisingly a QuorumListener. Quorum listeners are local to the node that they are registered, so they receive only events occurred on that local node. 3 com.company.quorum.ThreeNodeQuorumListener quorumRuleWithThreeNodes The QuorumListener has just one method that is called passing you aQuorumEvent. package com.hazelcast.quorum; import java.util.EventListener; /** * Listener to get notified when a quorum state is changed */ public interface QuorumListener extends EventListener { /** * Called when quorum presence state is changed. * * @param quorumEvent provides information about quorum presence and current member list. */ void onChange(QuorumEvent quorumEvent); } The QuorumEvent itself allows you to determine if a Quorum has been established or if it has been lost via its isPresent() method call. Additionally it provides the required cluster members to form a quorum and also the current membership list. Query the Quorums. Above we saw how we could receive callbacks, but in some cases we may just wish to make an immediate check to see if the Quorum is established or not. We can do this via the QuorumService. HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance(config); QuorumService quorumService = hazelcastInstance.getQuorumService(); Quorum quorum = quorumService.getQuorum(quorumName); boolean quorumPresence = quorum.isPresent(); In Conclusion The Cluster Quorum feature is another important tool for you to manage your cluster. In future versions of Hazelcast there are plans to add other data structures, for example you’ll be able to protect operations against Topics or Queues.
June 24, 2015
by Andrea Echstenkamper
· 2,830 Views · 2 Likes
  • Previous
  • ...
  • 556
  • 557
  • 558
  • 559
  • 560
  • 561
  • 562
  • 563
  • 564
  • 565
  • ...
  • Next
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook
×