Learn how to use the service registration and discovery services in ZooKeeper to manage microservices when refactoring from an existing monolithic application.
Distributed transaction tracing is a useful way to monitor or evaluate your microservices architecture for performance, particularly when measuring end-to-end requests.
In this article I will tell you how Apache Camel can turn a full-stack Linux microcomputer (like Raspberry Pi) into a device collecting the GPS coordinates.
written by tim brock. scatter plots are a wonderful way of showing ( apparent ) relationships in bivariate data. patterns and clusters that you wouldn't see in a huge block of data in a table can become instantly visible on a page or screen. with all the hype around big data in recent years it's easy to assume that having more data is always an advantage. but as we add more and more data points to a scatter plot we can start to lose these patterns and clusters. this problem, a result of overplotting, is demonstrated in the animation below. the data in the animation above is randomly generated from a pair of simple bivariate distributions. the distinction between the two distributions becomes less and less clear as we add more and more data. so what can we do about overplotting? one simple option is to make the data points smaller. (note this is a poor "solution" if many data points share exactly the same values.) we can also make them semi-transparent. and we can combine these two options: these refinements certainly help when we have ten thousand data points. however, by the time we've reached a million points the two distributions have seemingly merged in to one again. making points smaller and more transparent might help things; nevertheless, at some point we may have to consider a change of visualization. we'll get on to that later. but first let's try to supplement our visualization with some extra information. specifically let's visualize the marginal distributions . we have several options. there's far too much data for a rug plot , but we can bin the data and show histograms . or we can use a smoother option - a kernel density plot . finally, we could use the empirical cumulative distribution . this last option avoids any binning or smoothing but the results are probably less intuitive. i'll go with the kernel density option here, but you might prefer a histogram. the animated gif below is the same as the gif above but with the smoothed marginal distributions added. i've left scales off to avoid clutter and because we're only really interested in rough judgements of relative height. adding marginal distributions, particularly the distribution of variable 2, helps clarify that two different distributions are present in the bivariate data. the twin-peaked nature of variable 2 is evident whether there are a thousand data points or a million. the relative sizes of the two components is also clear. by contrast, the marginal distribution of variable 1 only has a single peak, despite coming from two distinct distributions. this should make it clear that adding marginal distributions is by no means a universal solution to overplotting in scatter plots. to reinforce this point, the animation below shows a completely different set of (generated) data points in a scatter plot with marginal distributions. the data again comes from a random sample of two different 2d distributions, but both marginal distributions of the complete dataset fail to highlight this separation. as previously, when the number of data points is large the distinction between the two clusters can't be seen from the scatter plot either. returning to point size and opacity, what do we get if we make the data points very small and almost completely transparent? we can now clearly distinguish two clusters in each dataset. it's difficult to make out any fine detail though. since we've lost that fine detail anyway, it seems apt to question whether we really want to draw a million data points. it can be tediously slow and impossible in certain contexts. 2d histograms are an alternative. by binning data we can reduce the number of points to plot and, if we pick an appropriate color scale, pick out some of the features that were lost in the clutter of the scatter plot. after some experimenting i picked a color scale that ran from black through green to white at the high end. note, this is (almost) the reverse of the effect created by overplotting in the scatter plots above. in both 2d histograms we can clearly see the two different clusters representing the two distributions from which the data is drawn. in the first case we can also see that there are more counts from the upper-left cluster than the bottom-right cluster, a detail that is lost in the scatter plot with a million data points (but more obvious from the marginal distributions). conversely, in the case of the second dataset we can see that the "heights" of the two clusters are roughly comparable. 3d charts are overused, but here (see below) i think they actually work quite well in terms of providing a broad picture of where the data is and isn't concentrated. feature occlusion is a problem with 3d charts so if you're going to go down this route when exploring your own data i highly recommend using software that allows for user interaction through rotation and zooming. in summary, scatter plots are a simple and often effective way of visualizing bivariate data. if, however, your chart suffers from overplotting, try reducing point size and opacity. failing that, a 2d histogram or even a 3d surface plot may be helpful. in the latter case be wary of occlusion.
Introduction Here in this article we are trying to discuss about the finding reference object within stored procedure and also finding the calling procedure references. Hope you like it and it will be informative. What We Want Developers are writing several stored procedure almost every day. Sometimes developers need to know about the information such as what object is used within the stored procedure or from where (SP) the specified stored procedure call. This is the vital information for the developer before working on a particular stored procedure. Here we are representing a pictorial diagram to understand the nature of implementation. Now we have to answer some question 1. What are the DB Object used in Stored Procedure1 and there type. 2. In case of Store Procedure3 which procedure calls the Store Procedure3 So we are not going to read the Stored Procedure to find the answer. Suppose the each procedure have more than 3000 line. How We Solve the Answer To solve the answer first we take the example and create an example scenario to understand it. -- Base Table CREATE TABLE T1 (EMPID INT, EMPNAME VARCHAR(50)); GO CREATE TABLE T2 (EMPID INT, EMPNAME VARCHAR(50)); GO --1 CREATE PROCEDURE [dbo].[Procedure1] AS BEGIN SELECT * FROM T1; SELECT * FROM T2; EXEC [dbo].[Procedure3]; END GO --2 CREATE PROCEDURE [dbo].[Procedure2] AS BEGIN EXEC [dbo].[Procedure3]; END GO --3 CREATE PROCEDURE [dbo].[Procedure3] AS BEGIN SELECT * FROM T1; END GO Now we are going to solve the question What are the DB Object used in Stored Procedure1 and there type. sp_depends Procedure1 In case of Store Procedure3 which procedure calls the Store Procedure3 SELECT OBJECT_NAME(id) AS [Calling SP] FROM syscomments WHERE [text] LIKE '%Procedure3%' GROUP BY OBJECT_NAME(id); Hope you like it.
in the previous blog post about solrcloud we’ve talked about the situation when zookeeper connection failed and how solr handles that situation. however, we only talked about query time behavior of solrcloud and we said that we will get back to the topic of indexing in the future. that future is finally here – let’s see what happens to indexing when zookeeper connection is not available. looking back at the old post in the solrcloud – what happens when zookeeper fails? blog post, we’ve shown that solr can handle querying without any issues when connection to zookeeper has been lost (which can be caused by different reasons). of course this is true until we change the cluster topology. unfortunately, in case of indexing or cluster change operations, we can’t change the cluster state or index documents when zookeeper connection is not working or zookeeper failed to read/write the data we want. why we can run queries? the situation is quite simple – querying is not an operation that needs to alter solrcloud cluster state. the only thing solr needs to do is accept the query, run it against known shards/replicas and gather the results. of course cluster topology is not retrieved with each query, so when there is no active zookeeper connection (or zookeeper failed) we don’t have a problem with running queries. there is also one important and not widely know feature of solrcloud – the ability to return partial results. by adding the shards.tolerant=true parameter to our queries we inform solr, that we can live with partial results and it should ignore shards that are not available. this means that solr will return results even if some of the shards from our collection is not available. by default, when this parameter is not present or set to false , solr will just return error when running a query against collection that doesn’t have all the shards available. why we can’t index data? so, we can’t we index data, when zookeeper connection is not available or when zookeeper doesn’t have a quorum? because there is potentially not enough information about the cluster state to process the indexing operation. solr just may not have the fresh information about all the shards, replicas, etc. because of that, indexing operation may be pointed to incorrect shard (like not to the current leader), which can lead to data corruption. and because of that indexing (or cluster change) operation is jus not possible. it is generally worth remembering, that all operations that can lead to cluster state update or collections update won’t be possible when zookeeper quorum is not visible by solr (in our test case, it will be a lack of connectivity of a single zookeeper server). of course, we could leave you with what we wrote above, but let’s check if all that is true. running zookeeper a very simple step. for the purpose of the test we will only need a single zookeeper instance which is run using the following command from zookeeper installation directory: bin/zkserver.sh start we should see the following information on the console: jmx enabled by default using config: /users/gro/solry/zookeeper/bin/../conf/zoo.cfg starting zookeeper ... started and that means that we have a running zookeeper server. starting two solr instances to run the test we’ve used the newest available solr version – the 5.2.1 when this blog post was published. to run two solr instances we’ve used the following command: bin/solr start -e cloud -z localhost:2181 solr asked us a few questions when it was starting and the answers where the following: number of instances: 2 collection name: gettingstarted number of shards: 2 replication count: 1 configuration name: data_driven_schema_configs cluster topology after solr started was as follows: let’s index a few documents to see that solr is really running, we’ve indexed a few documents by running the following command: bin/post -c gettingstarted docs/ if everything went well, after running the following command: curl -xget 'localhost:8983/solr/gettingstarted/select?indent=true&q=*:*&rows=0' we should see solr responding with similar xml: 0 38 *:* true 0 we’ve indexed our documents, we have solr running. let’s stop zookeeper and index data to stop zookeeper server we will just run the following command in the zookeeper installation directory: bin/zkserver.sh stop and now, let’s again try to index our data: bin/post -c gettingstarted docs/ this time, instead of data being written into the collection we will get an error response similar to the following one: posting file index.html (text/html) to [base]/extract simpleposttool: warning: solr returned an error #503 (service unavailable) for url: http://localhost:8983/solr/gettingstarted/update/extract?resource.name=%2fusers%2fgro%2fsolry%2f5.2.1%2fdocs%2findex.html&literal.id=%2fusers%2fgro%2fsolry%2f5.2.1%2fdocs%2findex.html simpleposttool: warning: response: 5033cannot talk to zookeeper - updates are disabled.503 as we can see, the lack of zookeeper connectivity resulted in solr not being able to index data. of course querying still works. turning on zookeeper again and retrying indexing will be successful, because solr will automatically reconnect to zookeeper and will start working again. short summary of course this and the previous blog post related to zookeeper and solrcloud are only touching the surface of what is happening when zookeeper connection is not available. a very good test that shows us data consistency related information can be found at http://lucidworks.com/blog/call-maybe-solrcloud-jepsen-flaky-networks/ . i really recommend it if you would like to know what will happen with solrcloud in various emergency situations.
In 2015, the middleware world focuses on two buzzwords: Docker and Microservices. Software vendors still sell products such as an Enterprise Service Bus (ESB) or Complex Event Processing (CEP) engines. How is this related? Docker is a fascinating technology to deploy and distribute modules (middleware, applications, services) quickly and easily. Most people agree that Docker will change the future of software development in the next years. I will do another blog post about how Docker is related to TIBCO and how you can deploy and distribute Microservices with Docker and TIBCO products such as TIBCO EMS and BusinessWorks 6 easily. Microservices is NOT a technology, but a software architecture style. Many people say that Microservices kill the Enterprise Service Bus (ESB) because Microservices use smart endpoints and dumb pipes. I had a talk at the Microservices Meetup in Munich in June 2015. Most attendees were surprised, why TIBCO shall be relevant for Microservices. I heard that question in several customer meetings, too. This was the main motivation for this talk. I want to share the slide deck and video recording of the talk with you… Abstract: Why use TIBCO for Microservices? Microservices are the next step after SOA: Services implement a limited set of functions. Services are developed, deployed and scaled independently. Continuous Integration and Continuous Delivery control deployments. This way you get shorter time to results and increased flexibility. Microservices have to be independent regarding build, deployment, data management and business domains. A solid Microservices design requires single responsibility, loose coupling and a decentralized architecture. A Microservice can to be closed or open to partners and public via APIs. This session discusses the requirements, best practices and challenges for creating a good Microservices architecture, and if this spells the end of the Enterprise Service Bus (ESB). Key messages of the talk: Microservices = SOA done right Integration is key for success – the product name does not matter Real time event correlation is the game changer Slide Deck from Microservices Meetup in Munich, Germany Here is the slide deck: Microservices = Death of the Enterprise Service Bus (ESB)? from Kai Wähner Video Recording on Youtube The session was recorded (thanks to the guys from AutoScout24). Here is the Youtube upload: https://youtu.be/wMDHUTmUsKg Looking forward to your feedback… Is the ESB dead or not? If no, what kind of ESB (or better said in 2015: Service Delivery Platform) do you use? If yes, how to you implement “ESB features” in your projects? “Simple” REST services and server-code under the hood, or how else?
This days I’m reading about Microservices. The idea is great. Instead of building a monolithic script using one language/framowork. We create isolated services and we build our application using those services (speaking HTTP between services and application). That’s means we’ll have several microservices and we need to use them, and maybe sometimes change one service with another one. In this post I want to build one small container to handle those microservices. Similar idea than Dependency Injection Containers. As we’re going to speak HTTP, we need a HTTP client. We can build one using curl, but in PHP world we have Guzzle, a great HTTP client library. In fact Guzzle has something similar than the idea of this post: Guzzle services, but I want something more siple. Imagine we have different services: One Silex service (PHP + Silex) use Silex\Application; $app = new Application(); $app->get('/hello/{username}', function($username) { return "Hello {$username} from silex service"; }); $app->run(); Another PHP service. This one using Slim framework use Slim\Slim; $app = new Slim(); $app->get('/hello/:username', function ($username) { echo "Hello {$username} from slim service"; }); $app->run(); And finally one Python service using Flask framework from flask import Flask, jsonify app = Flask(__name__) @app.route('/hello/') def show_user_profile(username): return "Hello %s from flask service" % username if __name__ == "__main__": app.run(debug=True, host='0.0.0.0', port=5000) Now, with our simple container we can use one service or another use Symfony\Component\Config\FileLocator; use MSIC\Loader\YamlFileLoader; use MSIC\Container; $container = new Container(); $ymlLoader = new YamlFileLoader($container, new FileLocator(__DIR__)); $ymlLoader->load('container.yml'); echo $container->getService('flaskServer')->get('/hello/Gonzalo')->getBody() . "\n"; echo $container->getService('silexServer')->get('/hello/Gonzalo')->getBody() . "\n"; echo $container->getService('slimServer')->get('/hello/Gonzalo')->getBody() . "\n"; And that’s all. You can see the project in my github account.
There are many, many different processes and requirements companies have for managing their database schemas. Some allow the application to directly manage them on startup, some require SQL scripts be executed by hand. Some have schemas that can differ across customers, some have only one database to deal with. For people who prefer to execute SQL themselves, Liquibase has always supported an “updateSQL” mode which does not update the database but instead outputs what would be run. This allows developers and DBAs to know exactly what will be ran and even make modifications as needed before actually executing the script. Before version 3.2, however, Liquibase required an active database connection for updateSQL. It used that connection to determine the SQL dialect to use and to query the DATABASECHANGELOG table to learn what changeSets have already been executed. Controlling updateSql SQL Syntax With version 3.2, Liquibase added a new “offline” mode. Instead of specifying a jdbc url such as “jdbc:mysql://localhost/lbcat” you can use “offline:mysql” or “offline:postgresql” which lets Liquibase know what dialect to use. For finer dialect control, you can specify parameters like “offline:mysql?version=3.4&caseSensitive=false Available dialect parameters: version: Standard X.Y.Z version of the database productName: String description of the database, like the JDBC driver would return catalog: String containing the name of the default top-level container ('database' in some databases 'schema' in others) caseSensitive: Boolean value specifying if the database is case sensitive or not Tracking History With CSV These parameters let Liquibase know what SQL to generate for each changeSet, but without an active database connection you cannot rely on the DATABASECHANGELOG table to track what changeSets have already been ran. Instead, offline mode uses a CSV file which mimics the structure of the DATABASECHANGELOG table. By default, Liquibase will use a file called “databasechangelog.csv” in the working directory, but it can be specified with a “changeLogFile” parameter such as “offline:mssql?changeLogFile=path/to/file.csv” It is up to you to ensure that the contents of the csv file match what is in the database. Running updateSQL automatically appends to the CSV file under the assumption that you will apply the SQL to the database. Since the csv file matches a particular database, it isn’t something you normally would store or share under version control because every database can (and probably will) be in a different state. If you do store the files in a central location, you will probably want to at least have a separate file for each database. By default, the SQL generated by updateSql in offline mode will still contain the standard DATABASECHANGELOG insert statements, so each database that you apply the SQL to will still have a correct DATABASECHANGELOG table. This means that you can switch between a direct-connection update and offline updateSQL as needed. It also means that you can also extract the current contents of the DATABASECHANGELOG table to a CSV file and use that as the file passed to the offline connection to ensure you have the right contents in the file. If you do not want the DATABASECHANGELOG table SQL included in updateSQL output, there is an “outputLiquibaseSql” parameter which can be passed in your offline url. Possible outputLiquibaseSql values: "none" will output no DATABASECHANGELOG statements "data_only" will output only INSERT INTO DATABASECHANGELOG statements "all" will output CREATE TABLE DATABASECHANGELOG if the csv file does not exist as well as INSERT statements (default value) Offline Snapshots The new 3.4.0 release of Liquibase expands offline support with a new “snapshot” parameter which can be passed to the offline url pointing to a saved database structure. Liquibase will use the snapshot anywhere it would have normally needed to read the current database state. This allows you to use preconditions and perform diff and diffChangeLog operations without an active connection and even between snapshots of the same database from different points in time. To create a snapshot of your live databases, use the “—snapshotFormat=json” parameter on the “snapshot” command. Command line example: $ liquibase --url=jdbc:mysql://localhost/lbcat snapshot --snapshotFormat=json > snapshot.json or $ liquibase --url=jdbc:mysql://localhost/lbcat –outputFile=path/to/output.json snapshot --snapshotFormat=json NOTE: currently only “json” is supported as a snapshotFormat. You can then use that file with your offline url and any snapshot operations will use it as the database state. liquibase –url=jdbc:mysql://localhost/lbcat –referenceUrl=offline:mysql?snapshot=path/to/snapshot.json diff will compare the stored snapshot with the current database state liquibase –url=offline:mysql?snapshot=path/to/snapshot.json diff –referenceUrl=offline:mysql?snapshot=path/to/older-snapshot.json diff will compare two snapshots liquibase –url=offline:mysql?snapshot=path/to/snapshot.json generateChangeLog will generate a changelog based on what is in the snapshot liquibase –url=jdbc:mysql://localhost/lbcat –referenceUrl=offline:mysql?snapshot=path/to/snapshot.json diffChangeLog will generate a changelog based on what is new in the real database compared to what is in the snapshot.
CORE
