Does MySQL work with Entity Framework 4.0? The answer is: yes, it works! I just put up one experimental project to play with MySQL and Entity Framework 4.0 and in this posting I will show you how to get MySQL data to EF. Also I will give some suggestions how to deploy your applications to hosting and cloud environments. MySQL stuff As you may guess you need MySQL running somewhere. I have MySQL installed to my development machine so I can also develop stuff when I’m offline. The other thing you need is MySQL Connector for .NET Framework. Currently there is available development version of MySQL Connector/NET 6.3.5 that supports Visual Studio 2010. Before you start download MySQL and Connector/NET: MySQL Community Server Connector/NET 6.3.5 If you are not big fan of phpMyAdmin then you can try out free desktop client for MySQL – HeidiSQL. I am using it and I am really happy with this program. NB! If you just put up MySQL then create also database with couple of table there. To use all features of Entity Framework 4.0 I suggest you to use InnoDB or other engine that has support for foreign keys. Connecting MySQL to Entity Framework 4.0 Now create simple console project using Visual Studio 2010 and go through the following steps. 1. Add new ADO.NET Entity Data Model to your project. For model insert the name that is informative and that you are able later recognize. Now you can choose how you want to create your model. Select “Generate from database” and click OK. 2. Set up database connection Change data connection and select MySQL Database as data source. You may also need to set provider – there is only one choice. Select it if data provider combo shows empty value. Click OK and insert connection information you are asked about. Don’t forget to click test connection button to see if your connection data is okay. If everything works then click OK. 3. Insert context name Now you should see the following dialog. Insert your data model name for application configuration file and click OK. Click next button. 4. Select tables for model Now you can select tables and views your classes are based on. I have small database with events data. Uncheck the checkbox “Include foreign key columns in the model” – it is damn annoying to get them away from model later. Also insert informative and easy to remember name for your model. Click finish button. 5. Define your classes Now it’s time to define your classes. Here you can see what Entity Framework generated for you. Relations were detected automatically – that’s why we needed foreign keys. The names of classes and their members are not nice yet. After some modifications my class model looks like on the following diagram. Note that I removed attendees navigation property from person class. Now my classes look nice and they follow conventions I am using when naming classes and their members. NB! Don’t forget to see properties of classes (properties windows) and modify their set names if set names contain numbers (I changed set name for Entity from Entity1 to Entities). 6. Let’s test! Now let’s write simple testing program to see if MySQL data runs through Entity Framework 4.0 as expected. My program looks for events where I attended. using(var context = new MySqlEntities()) { var myEvents = from e in context.Events from a in e.Attendees where a.Person.FirstName == "Gunnar" && a.Person.LastName == "Peipman" select e; Console.WriteLine("My events: "); foreach(var e in myEvents) { Console.WriteLine(e.Title); } } Console.ReadKey(); And when I run it I get the result shown on screenshot on right. I checked out from database and these results are correct. At first run connector seems to work slow but this is only the effect of first run. As connector is loaded to memory by Entity Framework it works fast from this point on. Now let’s see what we have to do to get our program work in hosting and cloud environments where MySQL connector is not installed. Deploying application to hosting and cloud environments If your hosting or cloud environment has no MySQL connector installed you have to provide MySQL connector assemblies with your project. Add the following assemblies to your project’s bin folder and include them to your project (otherwise they are not packaged by WebDeploy and Azure tools): MySQL.Data MySQL.Data.Entity MySQL.Web You can also add references to these assemblies and mark references as local so these assemblies are copied to binary folder of your application. If you have references to these assemblies then you don’t have to include them to your project from bin folder. Also add the following block to your application configuration file. ... ... Conclusion It was not hard to get MySQL connector installed and MySQL connected to Entity Framework 4.0. To use full power of Entity Framework we used InnoDB engine because it supports foreign keys. It was also easy to query our model. To get our project online we needed some easy modifications to our project and configuration files.

As promised I am going to provide an article on how we can use Sphinx with Java to perform a full text search. I will begin the article with an introduction to Sphinx. Introduction to Sphinx Databases are continually growing and sometimes tend to hold about 100M records and need an external solution for full text search to be performed. I have picked Sphinx, an open source full-text search engine, distributed under GPL version 2 to perform a full text search on such a huge amount of data. Generally, it's a standalone search engine meant to provide fast, size-efficient and relevant full-text search functions to other applications very much compatible with an SQL Database. So my example will be based on the MySQL database, as we cannot produce millions of data to evaluate the real power of Sphinx, we will have a small amount of data and I think that should not be a problem. Here are few Sphinx Unique Features: high indexing speed (up to 10 MB/sec on modern CPUs) high search speed (avg query is under 0.1 sec on 2-4 GB text collections) high scalability (up to 100 GB of text, upto 100 M documents on a single CPU) provides distributed searching capabilities provides searching from within MySQL through pluggable storage engine supports boolean, phrase, and word proximity queries supports multiple full-text fields per document (upto 32 by default) supports multiple additional attributes per document (ie. groups, timestamps, etc) supports MySQL natively (MyISAM and InnoDB tables are both supported) The important features which have been adopted to perform a full text search are the provision of the Java API to integrate easily with the web application and considerably high indexing and searching speed with an average of 4-10 MB/sec & 20-30 ms/q @5GB,3.5M docs(wikipedia) Sphinx Terms & How It Works The fist principle part of sphinx is indexer. It is solely responsible for gathering the data that will be searchable. From the Sphinx point of view, the data it indexes is a set of structured documents, each of which has the same set of fields. This is biased towards SQL, where each row corresponds to a document, and each column to a field. Sphinx builds a special data structure optimized for our queries from the data provided. This structure is called index; and the process of building index from data is called indexing and the element of sphinx which carries out these tasks is called indexer. Indexer can be executed either from a regular script or command-line interface. Sphinx documents are equal to records in DB. Document is set of text fields and number attributes + unique ID – similar to row in DB Set of fields and attributes is constant for index – similar to table in DB Fields are searchable for FullText queries Attributes may be used for filtering, sorting, grouping searchd is the second principle tools as part of Sphinx. It is the part of the system which actually handles searches; it functions as a server and is responsible for receiving queries, processing them and returning a dataset back to the different APIs for client applications. Unlike indexer, searchd is not designed to be run either from a regular script or command-line calling, but instead either as a daemon to be called from init.d (on Unix/Linux type systems) or to be called as a service (on Windows-type systems). I am going to focus on Windows environment so later I will show you how we can install sphinx on windows as a service. Finally search is one of the helper tools within the Sphinx package. Whereas searchd is responsible for searches in a server-type environment, search is aimed at testing the index quickly without building a framework to make the connection to the server and process its response. This will only be used for testing sphinx from command – line and with respect to application’s requirement; searchd service will be used to query the MySql Server with a pre created index. Installation on Windows So now we come to the part of installing Sphinx on Windows: Download Sphinx from the official Sphinx download site i.e http://sphinxsearch.com (I downloaded Win32 release binaries with MySQL support: sphinx-0.9.9-win32.zip) Unzip the file to some folder, I unzipped to C:\devel\sphinx-0.9.9-win32 and added the bin directory to the windows path variable Well Sphinx is installed. Nice, simple, easy. Later I will tell how to set up indexes and search. Sample Application Till now I guess the whole motto of this article is clear to you, let's move ahead to define our sample application. We all use the Address Book to search for people by using their name or e-mail address when we want to immediately address an e-mail message to a specific person, people, or distribution list. We also search for people by using other basic information, such as e-mail alias, office location, and telephone number etc. I think most of the people on this planet are quire familiar with this kind of search, so let's make outlook address book as our sample database schema. Most of the fields are mapped from microsoft outlook, the only additional column is date of joining so that we can filter our queries based on joining dates of the employees. The example that I am going to put forth will use Sphinx to search for a particluar address entry using free text search, meaning the user is free to type in anything, here is our search screen, the DOJ (date of joining) search parameter is optional. The screen is self explanatory, let's move ahead and define our database. As Sphinx works well with MySQL and MySQL being free also, lets create our db scripts around mysql database (Those who wish to install MySQL can dowload it from http://www.mysql.com) Let's create our sample database 'addressbook' mysql> create database addressbook; Query OK, 1 row affected (0.03 sec) mysql> use addressbook; Database changed Note: The fields defined in the following tables are for the purpose of learning only and may not contain a complete set of fields that microsoft address book or any similar software may provide. mysql> CREATE TABLE addressbook ( Id int(11) NOT NULL, FirstName varchar(30) NOT NULL, LastName varchar(30) NOT NULL, OfficeId int(11) DEFAULT NULL, Title varchar(20) DEFAULT NULL, Alias varchar(20) NOT NULL, Email varchar(50) NOT NULL, DOJ date NOT NULL, PhoneNo varchar(20) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8; mysql> CREATE TABLE CompanyLocations ( Id int(11) NOT NULL, Location varchar(60) NOT NULL, Country varchar(20) NOT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8; It's time to put some dummy data into the table, so let's fill our tables. Our virtual company 'gogs.it' has six offices across India and Singapore as defined in the following insert script. mysql> insert into CompanyLocations (Id, Location, Country) VALUES (1, 'Tower One, Harbour Front, Singapore', 'SG'); insert into CompanyLocations (Id, Location, Country) VALUES (2, 'DLF Phase 3, Gurgaon, India', 'IN'); insert into CompanyLocations (Id, Location, Country) VALUES (3, 'Hiranandani Gardens, Powai, Mumbai, India', 'IN'); insert into CompanyLocations (Id, Location, Country) VALUES (4, 'Hinjwadi, Pune, India', 'IN'); insert into CompanyLocations (Id, Location, Country) VALUES (5, 'Toll Post, Nagrota, Jammu, India', 'IN'); insert into CompanyLocations (Id, Location, Country) VALUES (6, 'Bani (Kathua), India', 'IN'); Now comes the real stuff... The data sphinx is going to index, let's populate that as well...wooooo mysql> INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (1,'Aabheer','Kumar',1,'Mr','u534','[email protected]','2008-9-3', '+911234599990'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (2,'Aadarsh','Gupta',6,'Mr','u668','[email protected]','2007-2-23','+911234599991'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (3,'Aachman','Singh',5,'Mr','u2766','[email protected]','2006-12-18','+911234599992'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (4,'Aadesh','Shrivastav',5,'Mr','u3198','[email protected]','2007-11-23','+911234599993'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (5,'Aadi','manav',1,'Mr','u2686','[email protected]','2010-7-20','+911234599994'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (6,'Aadidev','singh',4,'Mr','u572','[email protected]','2010-8-18','+911234599995'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (7,'Aafreen','sheikh',4,'Smt','u1092','[email protected]','2007-7-11','+911234599996'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (8,'Aakar','Sherpa',5,'Mr','u1420','[email protected]','2009-10-3','+911234599997'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (9,'Aakash','Singh',4,'Mrs','u2884','[email protected]','2008-6-11','+911234599998'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (10,'Aalap','Singhania',4,'Mrs','u609','[email protected]','2010-10-8','+911234599999'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (11,'Aandaleeb','mahajan',1,'Smt','u131','[email protected]','2010-10-21','+911234580001'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (12,'Mamata','kumari',5,'Sh','u2519','[email protected]','2009-6-12','+911234580002'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (13,'Mamta','sharma',6,'Smt','u4123','[email protected]','2009-2-8','+911234580003'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (14,'Manali','singh',6,'Mr','u1078','[email protected]','2008-6-14','+911234580004'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (15,'Manda','saxena',1,'Mrs','u196','[email protected]','2010-9-4','+911234580005'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (16,'Salila','shetty',3,'Miss','u157','[email protected]','2009-11-15','+911234580006'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (17,'Salima','happy',3,'Mrs','u3445','[email protected]','2006-7-14','+911234580007'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (18,'Salma','haik',5,'Sh','u4621','[email protected]','2008-6-23','+911234580008'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (19,'Samita','patil',3,'Smt','u3156','[email protected]','2006-6-7','+911234580009'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (20,'Sameena','sheikh',5,'Mrs','u952','[email protected]','2008-8-13','+911234580010'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (21,'Ranita','gupta',5,'Mrs','u2664','[email protected]','2008-10-20','+911234580011'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (22,'Ranjana','sharma',1,'Sh','u3085','[email protected]','2010-6-21','+911234580012'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (23,'Ranjini','singh',6,'Mrs','u4200','[email protected]','2007-4-13','+911234580013'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (24,'Ranjita','vyapari',2,'Smt','u1109','[email protected]','2008-1-22','+911234580014'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (25,'Rashi','gupta',6,'Mrs','u3492','[email protected]','2006-2-2','+911234580015'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (26,'Rashmi','sehgal',3,'Mr','u3248','[email protected]','2008-9-9','+911234580016'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (27,'Rashmika','sexy',1,'Mrs','u4599','[email protected]','2009-3-12','+911234580017'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (28,'Rasika','dulari',3,'Smt','u2089','[email protected]','2009-1-24','+911234580018'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (29,'Dilber','lover',6,'Mr','u4241','[email protected]','2007-10-11','+911234580019'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (30,'Dilshad','happy',1,'Mr','u1564','[email protected]','2007-4-8','+911234580020'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (31,'Dipali','lights',5,'Sh','u1127','[email protected]','2006-11-1','+911234580021'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (32,'Dipika','lamp',1,'Sh','u2271','[email protected]','2010-12-17','+911234580022'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (33,'Dipti','brightness',5,'Smt','u422','[email protected]','2010-9-25','+911234580023'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (34,'Disha','singh',3,'Sh','u4604','[email protected]','2006-5-2','+911234580024'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (35,'Maadhav','Krishna',1,'Miss','u2561','[email protected]','2007-11-6','+911234580025'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (36,'Maagh','month',5,'Miss','u874','[email protected]','2008-5-8','+911234580026'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (37,'Maahir','Skilled',4,'Mr','u3372','[email protected]','2007-8-4','+911234580027'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (38,'Maalolan','Ahobilam',5,'Mrs','u3498','[email protected]','2007-7-9','+911234580028'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (39,'Maandhata','King',1,'Smt','u2089','[email protected]','2009-9-3','+911234580029'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (40,'Maaran','Brave',2,'Miss','u4020','[email protected]','2008-4-5','+9112345606001'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (41,'Maari','Rain',2,'Sh','u3593','[email protected]','2007-12-5','+9112345606002'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (42,'Madan','Cupid',4,'Mrs','u795','[email protected]','2007-11-11','+9112345606003'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (43,'Madangopal','Krishna',3,'Sh','u438','[email protected]','2007-2-19','+9112345606004'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (44,'sahil','gogna',1,'Sh','u2273','[email protected]','2007-10-7','+9112345606005'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (45,'nikhil','gogna',2,'Mr','u1240','[email protected]','2009-9-14','+9112345606006'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (46,'amit','gogna',5,'Sh','u3879','[email protected]','2006-2-8','+9112345606007'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (47,'krishan','gogna',4,'Miss','u3632','[email protected]','2010-9-20','+9112345606008'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (48,'anil','kashyap',4,'Smt','u3939','[email protected]','2010-3-15','+9112345606009'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (49,'sunil','kashyap',5,'Mrs','u3493','[email protected]','2008-3-16','+9112345606010'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (50,'sandy','singh',6,'Mrs','u4691','[email protected]','2009-6-2','+9112345606011'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (51,'vishal','kapoor',3,'Mr','u1087','[email protected]','2010-5-13','+9112345606012'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (52,'bala','ji',5,'Mrs','u4762','[email protected]','2007-8-9','+9112345606013'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (53,'karan','sarin',4,'Miss','u3030','[email protected]','2008-4-8','+9112345606014'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (54,'abhishek','kumar',4,'Miss','u1093','[email protected]','2008-12-21','+9112345605001'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (55,'babu','the',1,'Miss','u1055','[email protected]','2008-7-2','+9112345506001'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (56,'sandeep','gainda',3,'Miss','u1320','[email protected]','2010-5-14','+9112345606301'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (57,'dheeraj','kumar',3,'Miss','u3685','[email protected]','2007-10-14','+9112345606091'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (58,'dharmendra','chauhan',1,'Smt','u3235','[email protected]','2008-8-1','+9112345806001'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (59,'max','alan',3,'Smt','u3465','[email protected]','2009-5-5','+9112345608011'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (60,'hidayat','khan',3,'Smt','u958','[email protected]','2007-11-18','+911234599101'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (61,'himnashu','singh',4,'Miss','u2027','[email protected]','2008-3-2','+911234599102'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (62,'dinesh','kumar',6,'Sh','u3233','[email protected]','2008-5-9','+911234599103'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (63,'toshi','prakash',1,'Mr','u3766','[email protected]','2010-9-17','+911234599104'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (64,'niti','puri',3,'Mr','u3575','[email protected]','2009-11-15','+911234599105'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (65,'pawan','tikki',3,'Sh','u3919','[email protected]','2006-3-19','+911234599106'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (66,'gaurav','sharma',2,'Sh','u413','[email protected]','2010-4-2','+911234599107'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (67,'himanshu','verma',2,'Mrs','u4732','[email protected]','2009-3-20','+911234599108'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (68,'priyanshu','verma',3,'Sh','u183','[email protected]','2010-8-12','+911234599109'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (69,'nitika','luthra',2,'Mrs','u4259','[email protected]','2010-7-12','+911234599110'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (70,'neeru','gogna',2,'Sh','u1633','[email protected]','2010-6-23','+91532110000'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (71,'bindu','gupta',1,'Sh','u1859','[email protected]','2006-11-10','+91532110001'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (72,'gurleen','bakshi',5,'Miss','u1423','[email protected]','2007-7-1','+91532110003'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (73,'rahul','gupta',3,'Sh','u1223','[email protected]','2009-8-11','+91532110004'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (74,'jagdish','salgotra',3,'Mr','u12','[email protected]','2008-5-19','+91532110005'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (75,'vikas','sharma',3,'Smt','u465','[email protected]','2006-6-2','+91532110006'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (76,'poonam','mahendra',2,'Sh','u1744','[email protected]','2009-12-2','+91532110007'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (77,'pooja','kulkarni',3,'Mrs','u1903','[email protected]','2008-10-6','+91532110008'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (78,'priya','mahajan',6,'Sh','u4205','[email protected]','2010-8-5','+91532110009'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (79,'manoj','zerger',1,'Mrs','u3369','[email protected]','2009-12-4','+91532110010'); INSERT INTO AddressBook(Id, FirstName, LastName, OfficeId, Title, Alias, Email, DOJ, PhoneNo) VALUES (80,'mohan','master',5,'Mr','u2841','[email protected]','2010-10-7','+91532110011'); Please note that above employee data is just a data *only data* I created using a small java programme using random number generators and reading some names file, so you may find titles getting messed up :( We next create a procedure that we will use from java to fetch records that we just inserted. DROP PROCEDURE IF EXISTS search_address_book; CREATE PROCEDURE search_address_book(IN address_ids VARCHAR(1000) ) BEGIN DECLARE search_address_query VARCHAR(2000) DEFAULT ''; SET address_ids = CONCAT('\'', REPLACE(address_ids, ',', '\',\''), '\''); SET search_address_query = CONCAT(search_address_query, ' select ab.Id as Id , ab.FirstName as FName, ab.LastName as LName, cl.Location as Location, ab.Title as Title, ab.Alias as Alias, ab.Email as Email, ab.DOJ as DOJ, ab.PhoneNo as PhoneNo ' ); SET search_address_query = CONCAT(search_address_query, ' from AddressBook ab left join CompanyLocations cl on ab.OfficeId=cl.Id '); SET search_address_query = CONCAT(search_address_query, ' where ab.id IN (', address_ids ,') '); SET @statement = search_address_query; PREPARE dynquery FROM @statement; EXECUTE dynquery; DEALLOCATE PREPARE dynquery; END; # To get records for ids 1, 6 and 7, we run following commands: call search_address_book('1,6,7'); Configuring Sphinx It turns out that it was not terribly difficult to setup sphinx, but I had a hard time finding instructions on the web, so I'll post my steps here. By default Sphinx looks for 'sphinx.co.in' configuration file to come with indexes and other stuff, lets create and define source and index for our sample application addressbook.conf (read between the lines) ############################################################################# ## data source definition ############################################################################# source addressBookSource { ## SQL settings for 'mysql' ## type = mysql # some straightforward parameters for SQL source types sql_host = localhost sql_user = root sql_pass = root sql_db = addressbook sql_port = 3306 # optional, default is 3306 # pre-query, executed before the main fetch query sql_query_pre = SET NAMES utf8 # main document fetch query, integer document ID field MUST be the first selected column sql_query = \ select ab.Id as Id , ab.FirstName as FName, ab.LastName as LName, cl.Location as Location, \ ab.Title as Title, ab.Alias as Alias, ab.Email as Email, UNIX_TIMESTAMP(ab.DOJ) as DOJ, ab.PhoneNo as PhoneNo \ from AddressBook ab left join CompanyLocations cl on ab.OfficeId=cl.Id sql_attr_timestamp = DOJ # document info query, ONLY for CLI search (ie. testing and debugging) , optional, default is empty must contain $id macro and must fetch the document by that id sql_query_info = SELECT * FROM AddressBook WHERE id=$id } ############################################################################# ## index definition ############################################################################# # local index example, this is an index which is stored locally in the filesystem index addressBookIndex { # document source(s) to index source = addressBookSource # index files path and file name, without extension, make sure you have this folder path = C:\devel\sphinx-0.9.9-win32\data\addressBookIndex # document attribute values (docinfo) storage mode docinfo = extern # memory locking for cached data (.spa and .spi), to prevent swapping mlock = 0 morphology = none # make sure this file exists exceptions =C:\devel\sphinx-0.9.9-win32\data\exceptions.txt enable_star = 1 } ############################################################################# ## indexer settings ############################################################################# indexer { # memory limit, in bytes, kiloytes (16384K) or megabytes (256M) # optional, default is 32M, max is 2047M, recommended is 256M to 1024M mem_limit = 32M # maximum IO calls per second (for I/O throttling) # optional, default is 0 (unlimited) # # max_iops = 40 # maximum IO call size, bytes (for I/O throttling) # optional, default is 0 (unlimited) # # max_iosize = 1048576 # maximum xmlpipe2 field length, bytes # optional, default is 2M # # max_xmlpipe2_field = 4M # write buffer size, bytes # several (currently up to 4) buffers will be allocated # write buffers are allocated in addition to mem_limit # optional, default is 1M # # write_buffer = 1M } ############################################################################# ## searchd settings ############################################################################# searchd { # hostname, port, or hostname:port, or /unix/socket/path to listen on listen = 9312 # log file, searchd run info is logged here # optional, default is 'searchd.log' log = C:\devel\sphinx-0.9.9-win32\data\log\searchd.log # query log file, all search queries are logged here # optional, default is empty (do not log queries) query_log = C:\devel\sphinx-0.9.9-win32\data\log\query.log # client read timeout, seconds # optional, default is 5 read_timeout = 5 # request timeout, seconds # optional, default is 5 minutes client_timeout = 300 # maximum amount of children to fork (concurrent searches to run) # optional, default is 0 (unlimited) max_children = 30 # PID file, searchd process ID file name # mandatory pid_file = C:\devel\sphinx-0.9.9-win32\data\log\searchd.pid # max amount of matches the daemon ever keeps in RAM, per-index # WARNING, THERE'S ALSO PER-QUERY LIMIT, SEE SetLimits() API CALL # default is 1000 (just like Google) max_matches = 1000 # seamless rotate, prevents rotate stalls if precaching huge datasets # optional, default is 1 seamless_rotate = 1 # whether to forcibly preopen all indexes on startup # optional, default is 0 (do not preopen) preopen_indexes = 0 } # --eof-- Once the configuration is done, its time to index our sql data, the command to use is 'indexer' as shown below. C:\devel\sphinx-0.9.9-win32\bin>indexer.exe --all --config C:\devel\sphinx-0.9.9-win32\addressbook.conf CONSOLE: Sphinx 0.9.9-release (r2117) Copyright (c) 2001-2009, Andrew Aksyonoff using config file 'C:\devel\sphinx-0.9.9-win32\addressbook.conf'... indexing index 'addressBookIndex'... collected 80 docs, 0.0 MB sorted 0.0 Mhits, 100.0% done total 80 docs, 5514 bytes total 0.057 sec, 96386 bytes/sec, 1398.43 docs/sec total 2 reads, 0.000 sec, 3.5 kb/call avg, 0.0 msec/call avg total 7 writes, 0.000 sec, 2.5 kb/call avg, 0.0 msec/call avg Note: As I told earlier that Sphinx creates 1 document for each row, as we had 80 rows in the database so a total of 80 docs are created. Time taken is also very very small, believe me I tried with half million rows and it took around 3-4 seconds :) cool isn't it? Once the index is up let's try to search few records, the utility command to perform search is 'search'. Ok Sphinx maharaj* please search for employee whose alias is u4732 C:\devel\sphinx-0.9.9-win32\bin>search.exe --config C:\devel\sphinx-0.9.9-win32\addressbook.conf u4732 CONSOLE: Sphinx 0.9.9-release (r2117) Copyright (c) 2001-2009, Andrew Aksyonoff using config file 'C:\devel\sphinx-0.9.9-win32\addressbook.conf'... index 'addressBookIndex': query 'u4732 ': returned 1 matches of 1 total in 0.001 sec displaying matches: 1. document=67, weight=1, doj=Fri Mar 20 00:00:00 2009 Id=67 FirstName=himanshu LastName=verma OfficeId=2 Title=Mrs Alias=u4732 [email protected] DOJ=2009-03-20 PhoneNo=+911234599108 words: 1. 'u4732': 1 documents, 1 hits words: 1. 'u4732': 1 documents, 1 hits As you can see above this is a unique record for Himanshu. Note: You see a lot of information for the result, this is because of following line in our configuration file sql_query_info = SELECT * FROM AddressBook WHERE id=$id If you want to see less columns you need to change the sql_query_info in configuration file. Let's try another search, sphinx maharaj* please tell me which all rows have gurleen or toshi in them. C:\devel\sphinx-0.9.9-win32\bin>search.exe --config C:\devel\sphinx-0.9.9-win32\addressbook.conf --any toshi gurleen CONSOLE: displaying matches: 1. document=63, weight=2, doj=Fri Sep 17 00:00:00 2010 Id=63 FirstName=toshi LastName=prakash OfficeId=1 Title=Mr Alias=u3766 [email protected] DOJ=2010-09-17 PhoneNo=+911234599104 2. document=72, weight=2, doj=Sun Jul 01 00:00:00 2007 Id=72 FirstName=gurleen LastName=bakshi OfficeId=5 Title=Miss Alias=u1423 [email protected] DOJ=2007-07-01 PhoneNo=+91532110003 Exactly two records were returned and this is what we were expecting. The following special operators and modifiers can be used when using the extended matching mode: operator OR: nikhil | sahil operator NOT: hello -sandy hello !sandy field search operator: @Email [email protected] For a complete set of search features , I advise you to go through http://sphinxsearch.com/docs/manual-0.9.9.html#searching link. Sphinx as Windows Service Now our main aim is to use sphinx with JAVA API, so let's move towards that now, before java can utilize the true power of Sphinx, we need to start 'searchd' as a windows service so that our java programme can connect to sphinx search engine. Let's install Sphinx as a windows service so that our java program can use this daemon service to query the index that we just created, the command is : C:\devel\sphinx-0.9.9-win32\bin>searchd.exe --install --config C:\devel\sphinx-0.9.9-win32\addressbook.conf --servicename --port 9312 SphinxSearch CONSOLE: Sphinx 0.9.9-release (r2117) Copyright (c) 2001-2009, Andrew Aksyonoff Installing service... Service 'SphinxSearch' installed succesfully. Well now the sphinx is ready to serve us on port 9312 Note: If you try to install Sphinx without admin rights, you may get following error messages. C:\devel\sphinx-0.9.9-win32\bin>searchd.exe --install --config C:\devel\sphinx-0.9.9-win32\addressbook.conf --servicename --port 9312 SphinxSearch CONSOLE: Installing service... FATAL: OpenSCManager() failed: code=5, error=Access is denied. Once done you can start the service as: c:\>sc start SphinxSearch (or alternatively from the services screen, start 'services.msc' in windows Run) If some how you want to delete the service , use c:\>sc delete SphinxSearch Let's create an adapter to fetch data from the database. package it.gogs.sphinx.util; import it.gogs.sphinx.AddressBoook; import it.gogs.sphinx.exception.AddressBookBizException; import it.gogs.sphinx.exception.AddressBookTechnicalException; import java.sql.CallableStatement; import java.sql.Connection; import java.sql.DriverManager; import java.sql.ResultSet; import java.sql.SQLException; import java.util.ArrayList; import java.util.List; import org.apache.log4j.Logger; /** * Adapter to fetch data from the database. * * @author Munish Gogna * */ public class AddressBookAdapter { private static Logger logger = Logger.getLogger(AddressBookAdapter.class); private AddressBookAdapter() { // use in static way.. } private static Connection getConnection() throws AddressBookTechnicalException { String userName = "root"; String password = "root"; String url = "jdbc:mysql://localhost/addressbook"; try { Class.forName("com.mysql.jdbc.Driver").newInstance(); return DriverManager.getConnection(url, userName, password); } catch (Exception e) { throw new AddressBookTechnicalException("could not get connection"); } } public static List getAddressBookList(List addressIds) throws AddressBookTechnicalException, AddressBookBizException { List addressBoookList = new ArrayList(); if (addressIds == null || addressIds.size() == 0){ logger.error("AddressIds was null or empty, returning empty list"); return addressBoookList; } Connection connection = null; CallableStatement callableStatement = null; try { connection = getConnection(); callableStatement = connection.prepareCall("{ call search_address_book(?)}"); callableStatement.setString(1, Utils.toCommaString(addressIds)); callableStatement.execute(); ResultSet resultSet = callableStatement.getResultSet(); prepareResults(resultSet, addressBoookList); connection.close(); } catch (SQLException e) { logger.error("Problem connecting MYSQL - " + e.getMessage()); throw new AddressBookTechnicalException(e.getMessage()); } catch (AddressBookTechnicalException e) { logger.error("Problem connecting MYSQL - " + e.getMessage()); throw e; } finally{ if(connection != null){ try { connection.close(); } catch (SQLException e) { logger.error("Problem closing conection - " + e.getMessage()); e.printStackTrace(); } } } return addressBoookList; } private static void prepareResults(ResultSet resultSet, List addressBoookList) throws SQLException { AddressBoook addressBoook; while (resultSet.next()) { addressBoook = new AddressBoook(); addressBoook.setAlias(resultSet.getString("Alias")); addressBoook.setEmail(resultSet.getString("Email")); addressBoook.setfName(resultSet.getString("FName")); addressBoook.setlName(resultSet.getString("LName")); addressBoook.setOfficeLocation(resultSet.getString("Location")); addressBoook.setPhoneNo(resultSet.getString("PhoneNo")); addressBoook.setTitle(resultSet.getString("Title")); addressBoook.setDateOfJoining(resultSet.getDate("DOJ")); addressBoook.setId(resultSet.getLong("Id")); addressBoookList.add(addressBoook); } } } Next we create the SphinxInstance that will parse the keywords and date range and provide us a list of Ids that matches the search. package it.gogs.sphinx.util; import it.gogs.sphinx.DateRange; import it.gogs.sphinx.SearchCriteria; import it.gogs.sphinx.api.SphinxClient; import it.gogs.sphinx.api.SphinxException; import it.gogs.sphinx.api.SphinxMatch; import it.gogs.sphinx.api.SphinxResult; import it.gogs.sphinx.exception.AddressBookBizException; import java.util.ArrayList; import java.util.Date; import java.util.List; import org.apache.log4j.Logger; /** * Instance that will parse our free text and provide the results. * * Note: Make sure that 'searchd' is up and running before you use this class * @author Munish Gogna * */ public class SphinxInstance { private static String SPHINX_HOST = "localhost"; private static String SPHINX_INDEX = "addressBookIndex"; private static int SPHINX_PORT = 9312; private static SphinxClient sphinxClient; private static Logger logger = Logger.getLogger(SphinxInstance.class); static { sphinxClient = new SphinxClient(SPHINX_HOST, SPHINX_PORT); } public static List getAddressBookIds(SearchCriteria criteria) throws AddressBookBizException, SphinxException { List addressIdsList = new ArrayList(); try { if (Utils.isNull(criteria)) { logger.error("criteria is null"); throw new AddressBookBizException("criteria is null"); } if (Utils.isNull(criteria.getKeywords())) { logger.error("keyword is a required field"); throw new AddressBookBizException("keyword is a required field"); } DateRange dateRange = criteria.getDateRage(); if (!Utils.isNull(dateRange)) { if (Utils.isDateRangeValid(dateRange)) { // this is to filter results based on joining dates if they are provided sphinxClient.SetFilterRange("DOJ", getTimeInSeconds(dateRange.getFromDate()), getTimeInSeconds(dateRange.getToDate()), false); } else { logger.error(" fromDate/toDate should not be empty and 'fromDate' should be less than equal to 'toDate'"); throw new AddressBookBizException("fromDate/toDate should not be empty and 'fromDate' should be less than equal to 'toDate'"); } } sphinxClient.SetMatchMode(SphinxClient.SPH_MATCH_EXTENDED2); sphinxClient.SetSortMode(SphinxClient.SPH_SORT_RELEVANCE, ""); SphinxResult result = sphinxClient.Query(buildSearchQuery(criteria), SPHINX_INDEX, "buidling query for address book search"); SphinxMatch[] matches = result.matches; for (SphinxMatch match : matches) { addressIdsList.add(String.valueOf(match.docId)); } } catch (SphinxException e) { throw e; } catch (AddressBookBizException e) { throw e; } logger.info("Total record(s):" + addressIdsList.size()); return addressIdsList; } private static long getTimeInSeconds(Date time) { return time.getTime()/1000; } private static String buildSearchQuery(SearchCriteria criteria) throws AddressBookBizException { String keywords[] = criteria.getKeywords().split(" "); StringBuilder searchFor = new StringBuilder(); for (String key : keywords) { if (!Utils.isEmpty(key)) { searchFor.append(key); if (searchFor.length() > 1) { searchFor.append("*|*"); } } } searchFor.delete(searchFor.lastIndexOf("|*"), searchFor.length()); StringBuilder queryBuilder = new StringBuilder(); String query = searchFor.toString(); queryBuilder.append("@FName *" + query + " | "); queryBuilder.append("@LName *" + query + " | "); queryBuilder.append("@Title *" + query + " | "); queryBuilder.append("@Location *"+ query + " | "); queryBuilder.append("@Alias *" + query + " | "); queryBuilder.append("@Email *" + query + " | "); queryBuilder.append("@PhoneNo *" + query); logger.info("Sphinx Query: " + queryBuilder.toString()); return queryBuilder.toString(); } } Here is the interface that I will expose to the outside world (in my future article I will expose this interface as Web Service) import it.gogs.sphinx.AddressBoook; import it.gogs.sphinx.SearchCriteria; import it.gogs.sphinx.api.SphinxException; import it.gogs.sphinx.exception.AddressBookBizException; import it.gogs.sphinx.exception.AddressBookTechnicalException; import java.util.List; /** * * @author Munish Gogna * */ public interface AddressBook { /** * Returns the list of AddressBook objects based on search criteria. * * @param criteria * @throws AddressBookTechnicalException * @throws AddressBookBizException * @throws SphinxException */ public List getAddressBookList(SearchCriteria criteria) throws AddressBookTechnicalException, AddressBookBizException, SphinxException; } and here is the implementation class for the same. package it.gogs.sphinx.addressbook.impl; import java.util.List; import it.gogs.sphinx.AddressBoook; import it.gogs.sphinx.SearchCriteria; import it.gogs.sphinx.addressbook.AddressBook; import it.gogs.sphinx.api.SphinxException; import it.gogs.sphinx.exception.AddressBookBizException; import it.gogs.sphinx.exception.AddressBookTechnicalException; import it.gogs.sphinx.util.AddressBookAdapter; import it.gogs.sphinx.util.SphinxInstance; /** * Implementation for our Address Book example * * @author Munish Gogna * */ public class AddressBookImpl implements AddressBook{ public List getAddressBookList(SearchCriteria criteria) throws AddressBookTechnicalException, AddressBookBizException, SphinxException { List addressIds= SphinxInstance.getAddressBookIds(criteria); return AddressBookAdapter.getAddressBookList(addressIds); } } ok so far so good, let's run some tests now ............ package it.gogs.sphinx.test; import java.util.Calendar; import java.util.GregorianCalendar; import java.util.List; import it.gogs.sphinx.AddressBoook; import it.gogs.sphinx.DateRange; import it.gogs.sphinx.SearchCriteria; import it.gogs.sphinx.addressbook.AddressBook; import it.gogs.sphinx.addressbook.impl.AddressBookImpl; import it.gogs.sphinx.api.SphinxException; import it.gogs.sphinx.exception.AddressBookBizException; import it.gogs.sphinx.exception.AddressBookTechnicalException; import junit.framework.TestCase; /** * * @author Munish Gogna * */ public class AddressBookTest extends TestCase { private AddressBook addressBook; @Override protected void setUp() throws Exception { super.setUp(); addressBook = new AddressBookImpl(); } @Override protected void tearDown() throws Exception { super.tearDown(); } /** this should be a unique record for Himanshu */ public void test_search_for_himanshu() throws Exception { SearchCriteria criteria = new SearchCriteria(); // remember the first 'search' example?? criteria.setKeywords("u4732"); List addressList = addressBook.getAddressBookList(criteria); assertTrue(addressList.size() == 1); assertTrue("expecting himanshu here", "himanshu".equals(addressList.get(0).getfName())); } /** only two employees have name gurleen or toshi */ public void test_search_for_gurleen_or_toshi() throws Exception { SearchCriteria criteria = new SearchCriteria(); // remember the second 'search' example?? criteria.setKeywords("gurleen toshi"); List addressList = addressBook.getAddressBookList(criteria); assertTrue(addressList.size() == 2); assertTrue("expecting toshi here", "toshi".equals(addressList.get(0).getfName())); assertTrue("expecting gurleen here", "gurleen".equals(addressList.get(1).getfName())); } /** there are 16 people from jammu location */ public void test_search_for_people_from_jammu_location() throws Exception { SearchCriteria criteria = new SearchCriteria(); criteria.setKeywords("jammu"); List addressList = addressBook.getAddressBookList(criteria); assertTrue(addressList.size() == 16); } /** only Aalap, Manda and nitika are having title as Mrs and joined in 2010 */ public void test_joined_in_2010_with_title_Mrs() throws Exception { DateRange dateRange = new DateRange(); GregorianCalendar calendar1 = new GregorianCalendar(); calendar1.set(Calendar.YEAR, 2010); calendar1.set(Calendar.MONTH, Calendar.JANUARY); calendar1.set(Calendar.DAY_OF_MONTH, 1); dateRange.setFromDate(calendar1.getTime()); GregorianCalendar calendar2 = new GregorianCalendar(); calendar2.set(Calendar.YEAR, 2010); calendar2.set(Calendar.MONTH, Calendar.DECEMBER); calendar2.set(Calendar.DAY_OF_MONTH, 31); dateRange.setToDate(calendar2.getTime()); SearchCriteria criteria = new SearchCriteria(); criteria.setKeywords("Mrs"); criteria.setDateRage(dateRange); List addressList = addressBook.getAddressBookList(criteria); assertTrue("expecting 3 records here", addressList.size() == 3); } /** should get a business exception here */ public void test_without_specifying_keywords(){ SearchCriteria criteria = new SearchCriteria(); //criteria.setKeywords("Mrs"); try { addressBook.getAddressBookList(criteria); } catch (Exception e) { assertTrue(e instanceof AddressBookBizException); assertTrue(e.getMessage().indexOf("keyword is a required field") >-1); } } } How we update the Index once database changes? For these kinds of requirements, we can set up two sources and two indexes, with one "main" index for the data which only changes rarely (if ever), and one "delta" for the new documents. First Time data will go in the "main" index and the newly inserted address book entries will go into "delta". Delta index could then be reindexed very frequently, and the documents can be made available to search in a matter of minutes. Also one thing to take from this article is once 'searchd' daemon is running we can't index the data in normal way,we have to use --rotate option in such cases. For some applications where there is a timely batch update for the data, we can configure some cron job to reindex our documents in Sphinx as shown below. C:\devel\sphinx-0.9.9-win32\bin>indexer.exe --all --config C:\devel\sphinx-0.9.9-win32\addressbook.conf --rotate Capsule We asked Sphinx to provide us the Document Ids corresponding to our search parameters and then we used those Ids to fire database query. In case the data we want to return is included in Index (DOJ attribute for example in our case) we can skip the database portion, so choose wisely how much information (attributes) you want to include while you index your sql data. Well that's all ... it's time to say good bye. Take good care of your health and don't forget to vote, its a must :) - Munish Gogna

In the previous article of this series, we saw how assertions are the key to self-validating tests. Even if an assertion's result is only a pass/fail value, programmers and testers find very handy to associate to the fail value some kind of additional information about the failure, to allow easier debugging without insertion of var_dump() and similar statements into the code. As I showed you in the previous article's code sample, assertions are useful the most when they fail, and in case they do, they should explain what happened so that a correction can be made quickly. An assertion that passes does not do anything; an assertion that fails shows an error. Whata happens when tests are failing The typical scenario is that of a large test suite, with hundreds or thousands of tests. At the time of a commit to the source code repository (or of a push if we're talking about git), the whole test suite must be green. Even if only one tests is red, the test suite is regarded as red. Sometimes, a regression is introduced by a programming error or an unforeseen scenario, and the output of the test suite becomes something like this: [12:30:02][giorgio@Desmond:~/txt/articles]$ phpunit assertionmessage.php PHPUnit 3.5.5 by Sebastian Bergmann. FFFF Time: 0 seconds, Memory: 3.75Mb There were 4 failures: 1) AssertionMessagesTest::testAsserTrue The object is not an Iterator. Failed asserting that is true. /home/giorgio/Dropbox/txt/articles/assertionmessage.php:9 2) AssertionMessagesTest::testAssertEquals The square root of 17 is not 4 but 4.1231056256177. Failed asserting that matches expected . /home/giorgio/Dropbox/txt/articles/assertionmessage.php:17 3) AssertionMessagesTest::testAssertContainsFailsWithCustomMessage The array does not contain 4. Failed asserting that an array contains . /home/giorgio/Dropbox/txt/articles/assertionmessage.php:24 4) AssertionMessagesTest::testAssertContains Failed asserting that an array contains . /home/giorgio/Dropbox/txt/articles/assertionmessage.php:36 FAILURES! Tests: 4, Assertions: 4, Failures: 4. The green tests are not noisy (which would be a smell), since they are not interesting at the moment. The red tests, which our focus is on, are shown with all the associated information. Along with the test name, the line of interest, and the kind of problem (failure/error), the assertion message is shown. How to write a meaningful message An assertion message can be passed as an argument of the assertion method, to be used in case of failure. Writing good assertion messages is an art, but before diving into some code to show you some messages I wrote, we can see how assertion messages are categorized in literature: Assertion-Identifying Message: includes information on which of the assertions in the same method caused the failure. This information can be for example the name of the checked variables. It's good to understand at a glance which assertion failed, but PHPUnit shows you the line of the test method that caused the failure, so it's not strictly necessary to include this kind of messages. Expectation-Describing Message: tells us what should have happened, but did not according to the data passed to the assertion method. Argument-Describing Message: when using assertions which are not very smart, like assertFalse() or assertTrue(), a failure would tell us by default only something like False was passed to assertTrue(). Adding an Argument-Describing Message will tell us instead The predicate on the set containing all red cars was false instead of true. Watch the test fail I can never say it enoug: when you write your tests, watch them fail (this mean also that each test should be written in advance with regard to the code it exercises, following the discipline of Test-Driven Development). This practice is not only useful for avoiding false positives, which are rare anyway, but to show at least for once the assertion message. You can check that when the test fails, the error message is meaningful and the test does not cause a Fatal Error. Watch your test fail to ensure that when they'll fail due to a regression in the future, you will be pleased by how fast you'll learn what's wrong. Example The code sample shows how to use PHPUnit's assertion methods with the $message optional argument. Any time you use basic methods like assertTrue(), a message is mandatory. The need for $message is inversely proportional to the specificity of the assertion: for example when assertContains() fails, PHPUnit has already valuable information in the the method itself to produce a readable error message. Note that the usage of "" (double quotes) enables fast composition of error messages via variable substitution. assertTrue($object instanceof Iterator, 'The object is not an Iterator.'); } public function testAssertEquals() { $expected = 4; $square = 17; $result = sqrt($square); $this->assertEquals($expected, $result, "The square root of $square is not $expected but $result."); } public function testAssertContainsFailsWithCustomMessage() { $array = array(1, 2, 3); $testElement = 4; $this->assertContains($testElement, $array, "The array does not contain $testElement."); } /** * This Assertion Message would be enough. It will even be better in some cases, * since PHPUnit would display nicely $testElement even if it was not "castable" * to a string (an object or array). */ public function testAssertContains() { $array = array(1, 2, 3); $testElement = 4; $this->assertContains($testElement, $array); } }

Some years back I had to provide a wrapper around an EJB 3.0 remote service to come up with a simple web service project that would be deployed over Tomcat and accessed in a simple http way due to some accessibility issues. Now as I cannot reveal the actual requirement I implemented that time so here I am presenting a simple demo kind of service with following signature. public AccountDetails getAccountDetails(String accountNo, SecurityToken token); The service will return the account details of a particular account number, provided the token is valid (generated using some Security module of the application). In nutshell, the client will ask for a token from Security module and then invoke this method. The service will validate the token to see if the caller can invoke the method or not? In general you should use handlers (message interceptors that can be easily plugged in to the JAX-WS runtime to do additional processing of the inbound and outbound messages) to validate the stuff, freeing implementation class from that overhead, it is just an example exercise so our implementation class will check the security token also.Sounds good.. lets move ahead. Libraries we are going to use include JAXB and JAX-WS, as both of them have sensible defaults, the number of annotations can be kept to the minimum. Also in my opinion, it is always best to develop WSDL and schemas by hand to ensure that the service contract is appropriately defined and also that the schema can be re-used (by other services) and extended if necessary. I do not prefer using annotations for automatically producing WSDL and schema at runtime so let's start with the definition, i am using inline schema as we have a very simple requirement here, though we should have schema definitions in separate xsd files. accounts.wsdl: I am not going to explain the In and outs of this wsdl , in a nutshell it defines one operation 'getAccountDetails' which takes accountNo and returns back the account details like account type, balance etc. Please note that I have also added a security header token that will validate the caller (left to the implementation of the service). Now as we are done with our wsdl and schema, let's generate the portable artifacts from our service definition. JAX-WS includes a tool that can do this for us, we will use this tool to generate portable artifacts like Service Endpoint Interface (SEI), Service and Exception classes.These artifacts can be packaged in a WAR file with the WSDL and schema documents along with the endpoint implementation to be deployed. To generate the artifacts , we run following command: wsimport C:\devel\workspace\webservice\WebContent\WEB-INF\wsdl\accounts.wsdl -p com.mg.ws -keep -Xnocompile This will create the following artifacts in com.mg.ws package. AccountDetails.java AccountDetailsFault.java AccountDetailsFault_Exception.java AccountDetailsPortType.java AccountDetailsTO.java GeAccountDetailsINTO.java ObjectFactory.java package-info.java SecurityToken.java Now as we have got all the artifacts we are ready to implement our service, the interface we need to implement is AccountDetailsPortType , so let's do it. Here is our dummy implementation class: package com.mg.ws.impl;import java.math.BigDecimal;import javax.jws.WebService;import com.mg.ws.AccountDetailsFault;import com.mg.ws.AccountDetailsFault_Exception;import com.mg.ws.AccountDetailsPortType;import com.mg.ws.AccountDetailsTO;import com.mg.ws.GeAccountDetailsINTO;import com.mg.ws.SecurityToken;@WebService(name = "AccountDetailsService", portName = "AccountDetailsPort", endpointInterface = "com.mg.ws.AccountDetailsPortType", wsdlLocation = "WEB-INF/wsdl/accounts.wsdl", targetNamespace="http://gognamunish.com/accounts") public class AccountDetailsServiceImpl implements AccountDetailsPortType {public AccountDetailsTO getAccountDetails(GeAccountDetailsINTO parameters, SecurityToken requestHeader) throws AccountDetailsFault_Exception { AccountDetailsTO detailsTO = new AccountDetailsTO(); // validate token validateToken(requestHeader); // populate response detailsTO = getDetailsFromSomewhere (parameters.getAccountNo()); return detailsTO; } private AccountDetailsTO getDetailsFromSomewhere(String accountNo) throws AccountDetailsFault_Exception { if(accountNo == null || accountNo.trim().length()==0){ AccountDetailsFault faultInfo = new AccountDetailsFault(); faultInfo.setFaultInfo("missing account number"); faultInfo.setMessage("account number is required field"); throw new AccountDetailsFault_Exception("account no missing", faultInfo); } AccountDetailsTO detailsTO = new AccountDetailsTO(); detailsTO.setAccNo(accountNo); detailsTO.setAccType("SAVING"); detailsTO.setBalance(new BigDecimal(10000)); return detailsTO; } private void validateToken(SecurityToken requestHeader) throws AccountDetailsFault_Exception { if ("83711070".equals(requestHeader.getToken()) && requestHeader.getValidTill() != null){ System.out.println("token processed successfully..."); } else { AccountDetailsFault faultInfo = new AccountDetailsFault(); faultInfo.setFaultInfo("Header token Invalid"); faultInfo.setMessage("can't help"); throw new AccountDetailsFault_Exception("invalid token", faultInfo); } } This is just a dummy implementation for illustration purpose only. Now as our service implementation is done, we proceed to package this in a war and deploy it on tomcat. Now we create a standard web.xml, which defines WSServletContextListener, WSServlet and structure of a web project. com.sun.xml.ws.transport.http.servlet.WSServletContextListener AccountDetailsServicecom.sun.xml.ws.transport.http.servlet.WSServlet 1AccountDetailsService/details Next we create a sun-jaxws.xml, defines the web service implementation class. This file is required regardless of whether we publish our web service on tomcat, glassfish or any other server. OK so far so good , let's build our application and deploy it on tomcat, here is the ant script. NOTE: jars in lib should be carefully chosen, otherwise it will make your life hell. Now let's test the application, point your browser to http://localhost:8080/account/details , if you see something like this, it means you have successfully deployed the service. Now let's test the service, we can use the client generated by the wsimport tool as : public static void main(String[] args) throws Exception { AccountDetails accountDetails = new AccountDetails(); AccountDetailsPortType port = accountDetails.getAccountDetailsPort(); AccountDetailsTO details = port.getAccountDetails(new GeAccountDetailsINTO(), new SecurityToken()); } For those who want to invoke the service using soap stuff, they can use tool like soapUI (can be downloaded from www.soapui.org), lets make some soap calls now: case: invalid security header case: valid security header By default, Tomcat does not comes with any JAX-WS dependencies, So, you have to include it manually. Go here Download JAX-WS RI distribution, you will find the wsimport tool in lib directory. Please note for running this project you can just skip this step as I have already done this step for you and included the required libraries in the lib folder of the project. I have included eclipse project for this demo, please get it from resource section. To deploy just change build.properties to point to TOMCAT_HOME and run ant build. Thats all for now ... Please provide your valuable comments or any suggestion and DON'T forget to vote :) It's Munish Gogna signing off now :)

Setting the mouse cursor position on a Windows machine with the help of .NET Framework shouldn't be that big of a problem. After all, there is the built-in Cursor class that lets you do that by executing a simple line of code: Cursor.Position = new System.Drawing.Point(0, 0); Of course, here 0 and 0 are the absolute coordinates for the mouse cursor on the screen. One thing to mention about this type of position setting is that Cursor requires a reference to System.Windows.Forms. And in some cases you don't want this extra reference. If that's the case, WinAPI is your solution. It requires some more work compared to the regular .NET way (class instance -> method call) but at the end you get more control than you would expect. When using WinAPI to set the cursor position, there are two ways you can go: mouse_event SendInput mouse_event is the very basic function that is only able to set the mouse coordinates. It was superseded and Microsoft recomends using SendInput instead. Nonetheless, it still works (although I cannot say for sure whether it will be working in future releases of Windows). So to start, I have a very basic class: class WINAPI_SUPERSEDED { [DllImport("user32.dll",SetLastError=true)] public static extern void mouse_event(uint dwFlags, uint dx, uint dy, uint dwData, int dwExtraInfo); public enum MouseFlags { MOUSEEVENTF_ABSOLUTE = 0x8000, MOUSEEVENTF_LEFTDOWN = 0x0002, MOUSEEVENTF_LEFTUP = 0x0004, MOUSEEVENTF_MIDDLEDOWN = 0x0020, MOUSEEVENTF_MIDDLEUP = 0x0040, MOUSEEVENTF_MOVE = 0x0001, MOUSEEVENTF_RIGHTDOWN = 0x0008, MOUSEEVENTF_RIGHTUP = 0x0010, MOUSEEVENTF_WHEEL = 0x0800, MOUSEEVENTF_XDOWN = 0x0080, MOUSEEVENTF_XUP = 0x0100 } public enum DataFlags { XBUTTON1 = 0x0001, XBUTTON2 = 0x0002 } } So to set the cursor position to 0,0 I would use this: WINAPI_SUPERSEDED.mouse_event((int)WINAPI_SUPERSEDED.MouseFlags.MOUSEEVENTF_MOVE | (int)WINAPI_SUPERSEDED.MouseFlags.MOUSEEVENTF_ABSOLUTE, 0, 0, 0, 0); If you go through the documentation, you will notice that in fact, dx and dy are in no way direct coordinates but rather normalized values ranged between 0 and 65,535 - that is only when the MOUSEEVENTF_ABSOLUTE flag is present. Otherwise, the position will be adjusted according to the current mouse cursor position. This method doesn't return any value so I cannot be informed whether it was successful or not. GetLastError won't give much information either. In this case, SendInput comes to the rescue. Here is what I have defined as the main class: class WINAPI { [DllImport("kernel32.dll")] public static extern uint GetLastError(); public enum MouseData { XBUTTON1 = 0x0001, XBUTTON2 = 0x0002 } public enum MouseFlags { MOUSEEVENTF_ABSOLUTE = 0x8000, MOUSEEVENTF_HWHEEL = 0x01000, MOUSEEVENTF_MOVE = 0x0001, MOUSEEVENTF_MOVE_NOCOALESCE = 0x2000, MOUSEEVENTF_LEFTDOWN = 0x0002, MOUSEEVENTF_LEFTUP = 0x0004, MOUSEEVENTF_RIGHTDOWN = 0x0008, MOUSEEVENTF_RIGHTUP = 0x0010, MOUSEEVENTF_MIDDLEDOWN = 0x0020, MOUSEEVENTF_MIDDLEUP = 0x0040, MOUSEEVENTF_VIRTUALDESK = 0x4000, MOUSEEVENTF_WHEEL = 0x0800, MOUSEEVENTF_XDOWN = 0x0080, MOUSEEVENTF_XUP = 0x0100 } [DllImport("user32.dll", SetLastError=true)] public static extern uint SendInput(uint nInputs, ref INPUT pInputs, int cbSize); [StructLayout (LayoutKind.Explicit)] public struct INPUT { [FieldOffset(0)] public int type; [FieldOffset(4)] public MOUSEINPUT mi; } public struct MOUSEINPUT { public int dx; public int dy; public int mouseData; public int dwFlags; public int time; public int extraInfo; } } This class is a bit more complicated, but at the same time you have to understand that in some cases, SendInput is used for hardware and keyboard input as well. Of course, for experimentation purposes I removed those parts in the sample class. Here you have the same MouseFlags enum that will let you pass custom flags defining the mouse behavior. Notice, that SendInput has SetLastError set to true, therefore if something wrong happens via this method, the error will be easily obtained via GetLastError, that is implemented as a helper method in the same class. VERY IMPORTANT: When you define the INPUT struct, make sure you use LayoutKind.Explicit since when passed to an unamanaged call, a specific field layout is required - as you can see, every field is decorated with a FieldOffset attribute. Also taking about StructLayout, you don't have to set StructLayout.Sequential to MOUSEINPUT since it is setautomatically by CLR. When I want to call the method above, I can simply use this snippet: WINAPI.MOUSEINPUT mouseInput = new WINAPI.MOUSEINPUT(); mouseInput.dx = 100; mouseInput.dy = 10; mouseInput.dwFlags = (int)WINAPI.MouseFlags.MOUSEEVENTF_ABSOLUTE | (int)WINAPI.MouseFlags.MOUSEEVENTF_MOVE; WINAPI.INPUT input = new WINAPI.INPUT(); input.type = 0; input.mi = mouseInput; uint x = WINAPI.SendInput(1, ref input, Marshal.SizeOf(input)); Console.WriteLine(x); Console.WriteLine(WINAPI.GetLastError()); Console.ReadLine(); Notice that I have to call the unmanaged version of sizeof and pass the INPUT struct to it in order for the method to correctly execute. The regular C# sizeof won't cut it here. When I am defining the type of input, 0 is repesenting the INPUT_MOUSE flag, since I am only handling the mouse here. Of course, I can re-organize my method to accept a set of INPUT instances - the native call itself allows this by requesting an array of INPUT and the correct indication of the number of INPUT instances passed, but that is not required for testing purposes.

Read about what a Thread Local is, and learn how to use it in this awesome tutorial.

Devoxx, and all similar conferences, is a place where you make new discoveries, continually. One of these, in my case, at last week's Devoxx, started from a discussion with Jaroslav Bachorik from the VisualVM team. He had presented VisualVM's extensibility in a session at Devoxx. I had heard that, when creating extensions for VisualVM, one can also create new charts using VisualVM's own charting API. Jaroslav confirmed this and we created a small demo together to prove it, i.e., there's a charting API in VisualVM. Since VisualVM is based on the NetBeans Platform, I went further and included the VisualVM charts in a generic NetBeans Platform application. Then I wondered what the differences are between JFreeChart and VisualVM charts, so asked the VisualVM chart architect, Jiri Sedlacek. He sent me a very interesting answer: JFreeCharts are great for creating any kind of static graphs (typically for reports). They provide support for all types of existing chart types. The benefit of using JFreeChart is fully customizable appearance and export to various formats. The only problem of this library is that it's not primarily designed for displaying live data. You can hack it to display data in real time, but the performance is poor. That's why I've created the VisualVM charts. The primary (and so far only) goal is to provide charts optimized for displaying live data with minimal performance and memory overhead. You can easily display a fullscreen graph and it will still scroll smoothly while running and adding new values (when running on physical hardware, virtualized environment may give slightly worse results). There's a real rendering engine behind the charts which ensures that only the changed areas of the chart are repainted (no full-repaints because of a 1px change). Scrolling the chart means moving the already rendered image and only painting the newly displayed area. Last but not least, the charts are optimized for displaying over a remote X session - rendering is automatically switched to low-quality ensuring good response times and interactivity. The Tracer engine introduced in VisualVM 1.3 further improves performance of the charts. I've intensively profiled and optimized the charts to minimize the cpu cycles/memory allocations for each repaint. As of now, I believe that the VisualVM charts are the fastest real time Java charts with the lowest cpu/memory footprint. Best of all is that everything described above is in the JDK. That's because VisualVM is in the JDK. Here's a small NetBeans Platform application (though you could also use the VisualVM chart API without using the NetBeans Platform, just include these JARs on your classpath: org-netbeans-lib-profiler-charts.jar, com-sun-tools-visualvm-charts.jar, com-sun-tools-visualvm-uisupport.jar and org-netbeans-lib-profiler-ui.jar) that makes use of the VisualVM chart API outlined above: The chart that you see above is updated in real time and you can change to full screen and you can scroll through it and, at the same time, there is no lag and it is very performant. Below is all the code (from the unit test package in the VisualVM sources) that you see in the JPanel above: public class Demo extends JPanel { private static final long SLEEP_TIME = 500; private static final int VALUES_LIMIT = 150; private static final int ITEMS_COUNT = 8; private SimpleXYChartSupport support; public Demo() { createModels(); setLayout(new BorderLayout()); add(support.getChart(), BorderLayout.CENTER); } private void createModels() { SimpleXYChartDescriptor descriptor = SimpleXYChartDescriptor.decimal(0, 1000, 1000, 1d, true, VALUES_LIMIT); for (int i = 0; i < ITEMS_COUNT; i++) { descriptor.addLineFillItems("Item " + i); } descriptor.setDetailsItems(new String[]{"Detail 1", "Detail 2", "Detail 3"}); descriptor.setChartTitle("Demo Chart"); descriptor.setXAxisDescription("X Axis [time]"); descriptor.setYAxisDescription("Y Axis [units]"); support = ChartFactory.createSimpleXYChart(descriptor); new Generator(support).start(); } private static class Generator extends Thread { private SimpleXYChartSupport support; public void run() { while (true) { try { long[] values = new long[ITEMS_COUNT]; for (int i = 0; i < values.length; i++) { values[i] = (long) (1000 * Math.random()); } support.addValues(System.currentTimeMillis(), values); support.updateDetails(new String[]{1000 * Math.random() + "", 1000 * Math.random() + "", 1000 * Math.random() + ""}); Thread.sleep(SLEEP_TIME); } catch (Exception e) { e.printStackTrace(System.err); } } } private Generator(SimpleXYChartSupport support) { this.support = support; } } } Here is the related Javadoc. To get started using the VisualVM charts in your own application, read this blog, and then look in the "lib" folder of the JDK to find the JARs you will need. And then have fun with real-time data in your Java desktop applications.

When you migrate an application using Apache Axis 1.2 from Java 4 or 5 to Java 6 (JRE 1.6) you will most likely encounter a handful of strange SOAP/SAAJ/XML errors and ClassCastExceptions. This is due to the fact that Sun’s implementation of SAAJ 1.3 has been integrated directly into the 1.6 JRE. Due to this integration it’s loaded by the bootstrap class loader and thus cannot see various classes that you might be referencing in your old code. As mentioned on Spring pages: Java 1.6 ships with SAAJ 1.3, JAXB 2.0, and JAXP 1.4 (a custom version of Xerces and Xalan). Overriding these libraries by putting different version on the classpath will result in various classloading issues, or exceptions in org.apache.xml.serializer.ToXMLSAXHandler. The only option for using more recent versions is to put the newer version in the endorsed directory (see above). Fortunately, there is a simple solution, at least for Axis 1.2. Some of the exceptions that we’ve encountered Sample Axis code import javax.xml.messaging.URLEndpoint;import javax.xml.soap.MessageFactory;import javax.xml.soap.SOAPConnection;import javax.xml.soap.SOAPConnectionFactory;import javax.xml.soap.SOAPMessage;...public static callAxisWebservice() {SOAPConnectionFactory soapconnectionfactory = SOAPConnectionFactory.newInstance();SOAPConnection soapconnection = soapconnectionfactory.createConnection();MessageFactory messagefactory = MessageFactory.newInstance();SOAPMessage soapmessage = messagefactory.createMessage();...URLEndpoint urlendpoint = new URLEndpoint(string);SOAPMessage soapmessage_18_ = soapconnection.call(soapmessage, urlendpoint);...} SOAPExceptionImpl: Bad endPoint type com.sun.xml.internal.messaging.saaj.SOAPExceptionImpl: Bad endPoint type http://example.com/ExampleAxisService at com.sun.xml.internal.messaging.saaj.client.p2p.HttpSOAPConnection.call(HttpSOAPConnection.java:161) This extremely confusing error is caused by the following, seemingly innocent code above, namely by the ‘… new URLEndpoint(string)’ and the call itself. The problem here is that Sun’s HttpSOAPConnection can’t see the javax.xml.messaging.URLEndpoint because it is not part of the JRE and is contained in another JAR, not visible to the classes loaded by the bootstrap loader. If you check the HttpSOAPConnection’s code (this is not exactly the version I have but close enough) you will see that it calls “Class.forName(“javax.xml.messaging.URLEndpoint”);” on line 101. For the reason mentioned it fails with a ClassNotFoundException (as indicated by the log “URLEndpoint is available only when JAXM is there” when you enable the JDK logging for the finest level) and thus the method isn’t able to recognize the type of the argument and fails with the confusing Bad endPoint message. A soluti0n in this case would be to pass a java.net.URL or a String instead of a URLEndpoint (though it might lead to other errors, like the one below). Related: Oracle saaj:soap1.2 bug SOAPExceptionImpl: Bad endPoint type. DOMException: NAMESPACE_ERR org.w3c.dom.DOMException: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces. at org.apache.xerces.dom.AttrNSImpl.setName(Unknown Source) at org.apache.xerces.dom.AttrNSImpl.(Unknown Source) at org.apache.xerces.dom.CoreDocumentImpl.createAttributeNS(Unknown Source) I don’t rembember exactly what we have changed on the classpath to get this confusing exception and I’ve no idea why it is thrown. Bonus: Conflict between Axis and IBM WebSphere JAX-RPC “thin client” Additionally, if you happen to have com.ibm.ws.webservices.thinclient_7.0.0.jar somewhere on the classpath, you may get this funny exception: java.lang.ClassCastException: org.apache.axis.Message incompatible with com.ibm.ws.webservices.engine.Message at com.ibm.ws.webservices.engine.soap.SOAPConnectionImpl.call(SOAPConnectionImpl.java:198) You may wonder why Java tries to use Axis Message with WebSphere SOAP connection. Well, it’s because the SAAJ lookup mechanism prefers the websphere implementation, for it declares itself via META-INF/services/javax.xml.soap.SOAPFactory pointing to com.ibm.ws.webservices.engine.soap.SOAPConnectionFactoryImpl, but instantiates the org.apache.axis.soap.MessageFactoryImpl for message creation for the websphere thin client doesn’t provide an implementation of this factory. The solution here is the same as for all the other exception, to use exclusively Axis. But if you are interested, check the description how to correctly create a Message with the websphere runtime on page 119 of the IBM WebSphere Application Server V7.0 Web Services Guide (md = javax.xml.ws.Service.create(serviceName).createDispatch(portName, SOAPMessage.class, Service.Mode.MESSAGE); ((SOAPBinding) ((BindingProvider) md).getBinding()).getMessageFactory();). Solution The solution that my collegue Jan Nad has found is to force JRE to use the SOAP/SAAJ implementation provided by Axis, something like: java -Djavax.xml.soap.SOAPFactory=org.apache.axis.soap.SOAPFactoryImpl -Djavax.xml.soap.MessageFactory=org.apache.axis.soap.MessageFactoryImpl -Djavax.xml.soap.SOAPConnectionFactory=org.apache.axis.soap.SOAPConnectionFactoryImpl example.MainClass It’s also described in issue AXIS-2777. Check details of the lookup process in the SOAPFactory.newInstance() JavaDoc. From http://theholyjava.wordpress.com/2010/11/19/soapsaajxml-issues-when-migrating-to-java-6-with-axis-1-2/

this tutorial shows you how to create a java web start (jnlp) file for user download. when the user clicks on the downloaded jnlp file, it launches a simple awt program. here's the summary steps : create a simple awt program and jar it as testjnlp.jar add keystore into testjnlp.jar create a jnlp file put it all into the tomcat folder access testjnlp.jar from web through http://localhost:8080/test.jnlp before starting this tutorial, lets read this brief java web start explanation from oracle. java web start is a mechanism for program delivery through a standard web server. typically initiated through the browser, these programs are deployed to the client and executed outside the scope of the browser. once deployed, the programs do not need to be downloaded again, and they can automatically download updates on startup without requiring the user to go through the whole installation process again. ok, let's go ~ 1. install jdk and tomcat install java jdk/jre version above 1.5 and tomcat. 2. directory structure directory structure of this example. 3. awt + jnlp see the content of testjnlp.java, it's just a simple awt program with jnlp supported. package com.mkyong;import java.awt.*;import javax.swing.*;import java.net.*;import javax.jnlp.*;import java.awt.event.actionlistener;import java.awt.event.actionevent;public class testjnlp { static basicservice basicservice = null; public static void main(string args[]) { jframe frame = new jframe("mkyong jnlp unofficial guide"); frame.setdefaultcloseoperation(jframe.exit_on_close); jlabel label = new jlabel(); container content = frame.getcontentpane(); content.add(label, borderlayout.center); string message = "jnln hello word"; label.settext(message); try { basicservice = (basicservice) servicemanager.lookup("javax.jnlp.basicservice"); } catch (unavailableserviceexception e) { system.err.println("lookup failed: " + e); } jbutton button = new jbutton("http://www.mkyong.com"); actionlistener listener = new actionlistener() { public void actionperformed(actionevent actionevent) { try { url url = new url(actionevent.getactioncommand()); basicservice.showdocument(url); } catch (malformedurlexception ignored) { } } }; button.addactionlistener(listener); content.add(button, borderlayout.south); frame.pack(); frame.show(); } p.s if "import javax.jnlp.*;" is not found, please include the jnlp library which islocated at jre/lib/javaws.jar. 4. jar it located your java classes folder. jar it with following command in command prompt jar -cf testjnlp.jar *.* this will package all the java's classes into a new jar file, named " testjnlp.jar ". 5. create keystore add a new keystore named " testkeys " keytool -genkey -keystore testkeys -alias jdc it will ask for a keystore password, first name, last name , organization's unit...etc..just fill them all. 6. assign keystore to jar file attached newly generated keystore " testkeys " to your " testjnlp.jar " file jarsigner -keystore testkeys testjnlp.jar jdc it will ask password for your newly created keystore 7. deploy jar it copy " testjnlp.jar " to tomcat's default web server folder, for example, in widnows - c:\program files\apache\tomcat 6.0\webapps\root 8. create jnlp file create a new test.jnlp file, content put this yong mook kim testing testing 9. deploy jnlp file copy test.jnlp to your tomcat default web server folder also. c:\program files\apache\tomcat 6.0\webapps\root 10. start tomcat start tomcat , c:\tomcat folder\bin\tomcat6.exe 11. test it access url http://localhost:8080/test.jnlp , it will prompt you to download the test.jnlp file, just accept and double click on it. if everything went fine, you should see the following output click on the "run" button to lauch the awt program. note if jnlp has no response, put the following code in your web.xml , which is located in the tomcat conf folder. jnlp application/x-java-jnlp-file

create a soap ui project using your wsdl. set the preferences in soap ui for axis2 home directory. right click on the wsdl in soap ui and click generate code. select adb binding and the following settings and click generate following is the directory structure and code files generated. that’s it, you can now use this code from you ide by importing it. ps: you will need to add the axis2 jars to your project class path. for more details visit my blog @ http://nitinaggarwal.wordpress.com/

Have you ever had to create a mock object in which most methods do nothing and are not called, but in others something useful needs to be done? EasyMock has some newish functionality to let you stub individual methods. But before I had heard about that, I had built a little framework (one base class) for creating mock objects which stubs those methods you want to stub, as well as logging every call made to the classes being mocked. It works like this: you choose a class which you need to mock, for example a service class called FooService, and you create a new class called FooServiceMock. You make it extend from AbstractMock, where T is the class you are mocking. As an example: public class FooServiceMock extends AbstractMock { public FooServiceMock() { super(FooService.class); } It needs to have a constructor to call the super constructor passing the class being mocked too. Perhaps that could be optimised, I don't have too much time right now. Next, you implement only those methods you expect to be called. For example: public class FooServiceMock extends AbstractMock { public FooServiceMock() { super(FooService.class); } /** * this is a method which exists in FooService, * but I want it to do something else. */ public String sayHello(String name){ return "Hello " + name + ", Foo here! This is a stub method!"; } To use the mock, you'll notice that it doesn't extend the class which it mocks, which might be problematic... Well, there are good reasons. To do the mocking, the abstract base class is actually going to create a dynamic proxy which wraps itself behind the interface of the class being mocked. To the caller, it looks like the FooService, but it's not actually anything related to it. Anytime a call to the FooService is made, the first thing which the proxy does is log that call, using XStream to create an XML representation of the parameters being passed into the method. Then, the proxy goes and looks in the instance of the mock class to see if it can find the method being called (well at least a method which takes the same parameters and has the same name and return type). If it finds such a method, it calls it. In our example, the sayHello(String) method would get called. It returns the result if there is one, to the caller. In the case where it cannot find the method, it throws an exception, because it assumes that if it was not implemented, you didn't expect it to be called. You could of course change this to suit your needs, maybe even calling the actual FooService. So, how to you use the FooServiceMock to create a FooService instance which you can use to mock your service? In the test, where you setup the class under test, you do this: FooServiceMock fooService = new FooServiceMock(); //perhaps tell it about objects you would //like it to return... instanceOfClassUnderTest.setFooService( fooService.getMock()); The setFooService(FooService) method on the instance of the class you are testing is in my case present, but you might not have it and may need to use reflection to do it. It's a question of how testable you write your classes, and is a design choice. The getMock() method on the AbstractMock class is the method which creates the dynamic proxy which wraps the instance of the mock. You can now test the class. There is however still something useful you can do after testing, i.e. assert that the right calls were made in the correct order with the right parameters. You do this in the test class to: assertEquals(1, fooService.getCalls().size()); assertEquals("[sayHello: Ant]", fooService.getCalls().toString()); The above tests that the sayHello(String) method was called just once, and passed the name "Ant". There are times when you might want to clear the call log, between parts of the test. For that, call the clearCalls() method on the mock object: fooService.clearCalls(); Have fun! From http://blog.maxant.co.uk/pebble/2010/11/03/1288813500000.html

Specification pattern is simple and clean solution to implement your business rules. This pattern is mixture between composite and factory Gof patterns introduced by Eric Evans & Martin Fowler. In this blog, we will not explain how this pattern works, but you can refer to wikipedia and Xebia for more details. But, we try here to translate specification pattern using the Scala language. Implementation : package me.jtunisie.specifications trait TSpecification [T]{ def ->(candidate :T ):Boolean def ||( specification : TSpecification[T]):TSpecification[T] def &&(specification :TSpecification[T] ):TSpecification[T] def unary_!():TSpecification[T] //Addition def | ( specification : TSpecification[T]):TSpecification[T] def & (specification :TSpecification[T] ):TSpecification[T] } The last two methods has been added compared to the specification. Sometimes, we need to do full check as A || B is not always equal to B || A. {Take as example true || (1/0 == 0 ). "We have encountered this problem when we validate an old embedded compiler written in C and in some platform like Solaris ( A || B ) starts by checking B and then A. } abstract class ASpecification[T] extends TSpecification [T]{ def ->(candidate :T ):Boolean def ||( s : TSpecification[T]):TSpecification[T] = OrSpecification(this,s) def &&(s :TSpecification[T] ):TSpecification[T] = AndSpecification(this,s) def unary_!():TSpecification[T] = NotSpecification(this) // Recurssive Add-0ns def | ( s : TSpecification[T]):TSpecification[T] = ROrSpecification(this,s) def &(s :TSpecification[T] ):TSpecification[T] = RAndSpecification(this,s) } To implement recursive and not recursive behavior, we have used /: and \: . According to documentation : (z /: List(a, b, c)) (op) equal to op(op(op(z, a), b), c) and (List(a, b, c) :\ z) (op) equal to op(a, op(b, op(c, z))) case class AndSpecification[T]( s: TSpecification[T]* ) extends ASpecification[T]{ override def -> (candidate :T ):Boolean= (true /: s) (_ -> candidate && _ -> candidate) } case class OrSpecification[T]( s: TSpecification[T]* ) extends ASpecification[T]{ override def ->(candidate :T ):Boolean= (false /: s) (_ -> candidate || _ -> candidate) } case class NotSpecification[T]( s: TSpecification[T] ) extends ASpecification[T]{ override def ->(candidate :T )= ! (s -> candidate ) } case class RAndSpecification[T]( s: TSpecification[T]* ) extends ASpecification[T]{ override def ->(candidate :T ):Boolean= (s :\ true) (_ -> candidate && _ -> candidate) } case class ROrSpecification[T]( s: TSpecification[T]* ) extends ASpecification[T]{ override def ->(candidate :T ):Boolean= (s :\ false) (_ -> candidate || _ -> candidate) } This code will not compile. In fact, Boolean class doesn't have -> method (IsSatisfiedBy). By Scala, we can use open class tool ( adding this method to Boolean class). Here is the implicit implementation : package me.jtunisie package object specifications { class MyBool(b:Boolean){ def ->(candidate :Any ):Boolean = b } implicit def toMyBool(b:Boolean)= new MyBool(b) } source code is under : http://github.com/ouertani/Rules Mode of use ( with precedence checked ): Object AlwaysOk extends ASpecification[Boolean] { override def -> (b: Boolean)= true } object AlwaysKo extends ASpecification[Boolean] { override def -> (b: Boolean)= false } object AlwaysOk extends ASpecification[Boolean] { override def -> (b: Boolean)= true } object AlwaysKo extends ASpecification[Boolean] { override def -> (b: Boolean)= false } false mustBe ((AlwaysOk || AlwaysKo) && AlwaysKo)-> (true) true mustBe (AlwaysOk || (AlwaysKo && AlwaysKo))-> (true) true mustBe ( AlwaysOk || AlwaysKo && AlwaysKo)-> (true)

MyBatis is a lightweight persistence framework for Java and .NET. This blog entry addresses the Java side. MyBatis is an alternative positioned somewhere between plain JDBC and ORM frameworks (e.g. EclipseLink or Hibernate). MyBatis usually uses XML, but it also supports annotations since version 3. The documentation is very detailed for XML, but lacks annotation examples. Just the Annotations itself are described, but no examples how to use them. I could not find any good and easy examples anywhere, so I will describe some very basic examples for SELECT, INSERT and UPDATE statements by implementing a Data Access Object (DAO) using MyBatis. These examples are a good starting point to create more complex MyBatis queries using a DAO. You can find the full source code at the end of this blog. A simple SQL-Table I use a very simple table with two attributes. The name of the table is “simple_information”. The primary key is a Integer and may not be null (info_id). The only real data is a character and may also not be null (info_content). That is enough “complexity” to learn the usage of MyBatis annotations. The Java class “SimpleInformationEntity” is a POJO which contains these two attributes. @SELECT-Statement The @Select annotation is very easy to use, if you want to use exactly one paramter. If you need more than one paramter, use the @Param annotation (which is described below at the update example). You do not have to map the found information to a SimpleInformationEntity object, as you would have to do with a JDBC ResultSet. The magic of the framework does this for you. final String GET_INFO = “SELECT * FROM simple_information WHERE info_id = #{info_id}”; @Select(GET_INFO) public SimpleInformationEntity getSimpleInformationById(int info_id) throws Exception; @INSERT-Statement You can use the object (which you want to be persist) as parameter. You do not have to use several parameters for each attribute of the object. The magic of the framework does this for you. final String PERSIST_INFO = “INSERT INTO simple_information(info_id, info_content) VALUES (#{infoId}, #{infoContent})”; @Insert(PERSIST_INFO) public int persistInformation(SimpleInformationEntity simpleInfo) throws Exception; @UPDATE-Statement You cannot use more than one parameter within a method. If you want to understand why, look at some MyBatis XML examples in the documenation: There you use the attribute “parameterType”, which must be exactly one “parameter”! So you will get a (strange) exception, if you use two or more parameters. Instead you have to use the @Param annotation, if you need more than one parameter. final String UPDATE_INFO = “UPDATE simple_information SET info_content = #{newInfo} WHERE info_id = #{infoId}”; @Update(UPDATE_INFO) public int updateInformation(@Param(“infoId”) int info_id, @Param(“newInfo”) String new_content) throws Exception; Configuration You have to add your MyBatis-Interface for the Mapper to the SQLSessionFactory: sqlSessionFactory.getConfiguration().addMapper(InfoMapper.class); Some Hints for developing with MyBatis The following means helped me a lot to use MyBatis annotations despite the lack of documentation about using annotations: - MyBatis is open source! So add the sources to your build path and use the debugging function of your IDE to enter the MyBatis source code while executing some queries. You will see what MyBatis expects as input and how it is processed. - Read the documentation about using MyBatis XML. This does not really make any sense you think? It does! The processing deep inside MyBatis does not change if you use annotations instead of XML. It is just another way to develop persistence queries. E.g. if you know that a XML select query may just use exactly one parameterType-attribute, then you know that you may just use one parameter in an annotation-based method too! If you need more parameters, you have to use the @Param annotation. - If you get any strange exception that does not make any sense, then clean and re-compile your project. This often helps, because as with other persistence frameworks such as Hibernate, the bytecode enhancement sometimes confuses your IDE. Conclusion: @MyBatis-Team: Improve and extend the documentation instead of improving the framework itself! MyBatis is a nice lightweigt persistence framework. But the documentation is not enough detailed. Some important information is completely missing. Especially, if you are a newbie to MyBatis / iBatis, it is very tough to develop with MyBatis using annotations instead of XML. Besides the usage of annotations, another good example for missing documentation is how to configure transactions in MyBatis by using JNDI and a J2EE / JEE Application server. You have to use google to find out, and if you are lucky you will find a mailing list or blog entry describing your problem. If not, you have to try it out. The missing documentation makes MyBatis much more tough than it actually is. So in the next months, the MyBatis team should improve and extend the documentation instead of improving the framework itself… Best regards, Kai Wähner (Twitter: @Kai Waehner) [Content from my Blog: MyBatis (formerly iBatis) - Examples and Hints using Select, Insert and Update Annotations - Kai Wähner's IT-Blog] Appendix: Source Code Here you see all the necessary source code, also including a MyBatis Connection Factory, which reads the configuration data from a XML file. ############################################################################ Connection Factory (using a static initializer) ############################################################################ import java.io.FileNotFoundException; import java.io.IOException; import java.io.InputStream; import java.io.InputStreamReader; import java.io.Reader; import org.apache.ibatis.session.SqlSessionFactory; import org.apache.ibatis.session.SqlSessionFactoryBuilder; import de.waehner.kai.persistence.InfoDAO.InfoMapper; public class MyBatisConnectionFactory { private static SqlSessionFactory sqlSessionFactory; static { Reader reader = null; try { InputStream in = MyBatisConnectionFactory.class.getResourceAsStream(“myBatisConfiguration.xml”); reader = new InputStreamReader(in); if (sqlSessionFactory == null) { sqlSessionFactory = new SqlSessionFactoryBuilder().build(reader); sqlSessionFactory.getConfiguration().addMapper(InfoMapper.class); } in.close(); } catch (FileNotFoundException fileNotFoundException) { fileNotFoundException.printStackTrace(); } catch (IOException iOException) { iOException.printStackTrace(); } } public static SqlSessionFactory getSqlSessionFactory() { return sqlSessionFactory; } } ############################################################################ Data Acces Object (including the MyBatis-Mapper as inner Class) ############################################################################ import org.apache.ibatis.annotations.Insert; import org.apache.ibatis.annotations.Param; import org.apache.ibatis.annotations.Select; import org.apache.ibatis.annotations.Update; import org.apache.ibatis.session.SqlSession; import org.apache.ibatis.session.SqlSessionFactory; public class InfoDAO { public interface InfoMapper { final String GET_INFO = “SELECT * FROM simple_information WHERE info_id = #{info_id}”; @Select(GET_INFO) public SimpleInformationEntity getSimpleInformationById(int info_id) throws Exception; final String PERSIST_INFO = “INSERT INTO simple_information(info_id, info_content) VALUES (#{infoId}, #{infoContent})”; @Insert(PERSIST_INFO) public int persistInformation(SimpleInformationEntity simpleInfo) throws Exception; final String UPDATE_INFO = “UPDATE simple_information SET info_content = #{newInfo} WHERE info_id = #{infoId}”; @Update(UPDATE_INFO) public int updateInformation(@Param(“infoId”) int info_id, @Param(“newInfo”) String new_content) throws Exception; } public SimpleInformationEntity getSingleAlarm(int info_id) throws Exception { SqlSessionFactory sqlSessionFactory = MyBatisConnectionFactory.getSqlSessionFactory(); SqlSession session = sqlSessionFactory.openSession(); try { InfoMapper mapper = session.getMapper(InfoMapper.class); SimpleInformationEntity simpleInfo = mapper.getSimpleInformationById(info_id); return simpleInfo; } finally { session.close(); } } public int persistInformation(SimpleInformationEntity simpleInfo) throws Exception { SqlSessionFactory sqlSessionFactory = MyBatisConnectionFactory.getSqlSessionFactory(); SqlSession session = sqlSessionFactory.openSession(); try { InfoMapper mapper = session.getMapper(InfoMapper.class); int answer = mapper.persistInformation(simpleInfo); return answer; } finally { session.close(); } } public int updateInformation(int info_id, String new_content) throws Exception { SqlSessionFactory sqlSessionFactory = MyBatisConnectionFactory.getSqlSessionFactory(); SqlSession session = sqlSessionFactory.openSession(); try { InfoMapper mapper = session.getMapper(InfoMapper.class); int answer = mapper.updateInformation(info_id, new_content); return answer; } finally { session.close(); } } } ############################################################################ SimpleInformation Entity (a simple POJO) ############################################################################ import java.io.Serializable; public class SimpleInformationEntity implements Serializable{ private static final long serialVersionUID = -821826330941829539L; private int infoId; private String infoContent; public int getInfoId() { return infoId; } public void setInfoId(int infoId) { this.infoId = infoId; } public String getInfoContent() { return infoContent; } public void setInfoContent(String infoContent) { this.infoContent = infoContent; } }

Shutdown Hooks are a special construct that allow developers to plug in a piece of code to be executed when the JVM is shutting down.

Yesterday I attended at local community evening where one of the most famous Estonian MVPs – Henn Sarv – spoke about SQL Server queries and performance. During this session we saw very cool demos and in this posting I will introduce you my favorite one – how to insert million numbers to table. The problem is: how to get one million numbers to table with less time? We can solve this problem using different approaches but not all of them are quick. Let’s go now step by step and see how different approaches perform. NB! The code samples here are not original ones but written by me as I wrote this posting. Using WHILE First idea for many guys is using WHILE. It is robust and primitive approach but it works if you don’t think about better solutions. Solution with WHILE is here. declare @i as int set @i = 0 while(@i < 1000000) begin insert into numbers values(@i) set @i += 1 end When we run this code we have to wait. Well… we have to wait couple of minutes before SQL Server gets done. On my heavily loaded development machine it took 6 minutes to run. Well, maybe we can do something. Using inline table As a next thing we may think that inline table that is kept in memory will boost up performance. Okay, let’s try out the following code. declare @t TABLE (number int) declare @i as int set @i = 0 while(@i < 1000000) begin insert into @t values(@i) set @i += 1 end insert into numbers select * from @t Okay, it is better – it took “only” 01:30 to run. It is better than six minutes but it is not good yet. Maybe we can do something more? Optimizing WHILE If we investigate the code in first example we can find one hidden resource eater. All these million inserts are run in separate transaction. Let’s try to run inserts in one transaction. declare @i as int set @i = 0 begin transaction while(@i < 1000000) begin insert into numbers values(@i) set @i += 1 end commit transaction Okay, it’s a lot better – 18 seconds only! Using only set operations Now let’s write some SQL that doesn’t use any sequential constructs like WHILE or other loops. We will write SQL that uses only set operations and no long running stuff like before. declare @t table (number int) insert into @t select 0 union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9 insert into numbers select t1.number + t2.number*10 + t3.number*100 + t4.number*1000 + t5.number*10000 + t6.number*100000 from @t as t1, @t as t2, @t as t3, @t as t4, @t as t5, @t as t6 Bad side of this SQL is that it is not as intuitive for application programmers as previous examples. But when you are working with databases you have to know how some set calculus as well. The result is now seven seconds! Results As last thing, let’s see the results as bar chart to illustrate difference between approaches. I think this example shows very well how usual optimization can give you better results but when you are moving to sets – this is something that SQL Server and other databases understand better – you can get very good results in performance.

This is the last article from the Practical PHP Patterns series. Stay tuned on css.dzone.com for the new series, Practical PHP Testing Patterns. The RecordSet pattern's goal is to represent a set of relational database rows, with the main purpose of giving access to their values (a data structure), sometimes with the possibility of modification (via single Row Data Gateway instances). Despite the term set, the rows have usually a defined order. The intent is ultimately representing a result from an SQL query with an object, to gain the usual advantages of objects over scalars and functions: it can be passed around but maintain its encapsulated behavior, injected, mocked, wrapped and so on. A Record Set is usually not mocked if it is provided by an external extension or library, because instancing a real one working on a lightweight database such as Sqlite is used in substitution for the real one. Especially in the PHP world, this solution is fast enough to become a standard. Today, Record Set is less used on the client side to favor Object-Relational Mapping approaches, which make some kind of translation over the raw rows (what an horrible pun). Record Set instead maintains by definition a one-to-one relationship with the table rows, and it is still diffused in ORM internals (such as Doctrine's own code) or in applications that call PDO directly. It is a fairly basic pattern, but given that a vast part of legacy PHP applications still uses mysql_query()... Interesting things happen when... Interesting leverages of this pattern happen when someone stays in the middle between the Record Set creation and the user interface, with the goal of modifying or decorating it. The UI can then explore the RecordSet and automatically generate itself, in a form of scaffolding. Continuing on this line of thought, UI components can edit the RecordSet without knowing the model which it refers to, via building forms driven by the Record Set metadata. This solution is diffused, but it does not scale to Domain Models with a level of complexity greater than plain arrays. Of course, the Record Set may also encapsulate business logic as a low-cost form of Domain Model. In this case, the ability to unlink it from the database connection is important to its serializability and ease of testing. Examples PDOStatement represents both a SQL query and a RecordSet implementation, after it has been executed. When fetching all the results, it returns an array. In other languages Record Sets are more evolved and can for example be used to navigate a table and modify only certain records (via their annexed Row Data Gateway). PDOStatement is used only for reading. If you want further functionalities (which obviously depends on your domain), you should create your own Record Set accordingly. It is probably best to wrap the PDOStatement because extending it is out of question due to the instantiation not being under our control. connection = $connection; } public function getTweetsRecordSet($username) { /** * @var PDOStatement this is a Record Set */ $stmt = $this->connection->prepare('SELECT * FROM tweets WHERE username = :username'); $stmt->bindValue(':username', $username, PDO::PARAM_STR); $stmt->execute(); return $stmt; } } $pdo = new PDO('sqlite::memory:'); $pdo->exec('CREATE TABLE tweets (id INT NOT NULL, username VARCHAR(255) NOT NULL, text VARCHAR(255) NOT NULL, PRIMARY KEY(id))'); $pdo->exec('INSERT INTO tweets (id, username, text) VALUES (42, "giorgiosironi", "Practical PHP Patterns has come to an end")'); $pdo->exec('INSERT INTO tweets (id, username, text) VALUES (43, "giorgiosironi", "Cool series: will continue as Practical PHP Testing Patterns")'); // client code $table = new TweetsTable($pdo); $recordSet = $table->getTweetsRecordSet('giorgiosironi'); while ($row = $recordSet->fetch()) { var_dump($row['text']); }

When I am writing enumerations I very often found myself implementing a static method similar to the standard enum’s valueOf() but based on field rather than name: public static TestOne valueOfDescription(String description) { for (TestOne v : values()) { if (v.description.equals(description)) { return v; } } throw new IllegalArgumentException( "No enum const " + TestOne.class + "@description." + description); } Where “description” is yet another String field in my enum. And I am not alone. See this article for example. Obviously this method is very ineffective. Every time it is invoked it iterates over all members of the enum. Here is the improved version that uses a cache: private static Map map = null; public static TestTwo valueOfDescription(String description) { synchronized(TestTwo.class) { if (map == null) { map = new HashMap(); for (TestTwo v : values()) { map.put(v.description, v); } } } TestTwo result = map.get(description); if (result == null) { throw new IllegalArgumentException( "No enum const " + TestTwo.class + "@description." + description); } return result; } It is fine if we have only one enum and only one custom field that we use to find the enum value. But if we have 20 enums, and each has 3 such fields, then the code will be very verbose. As I dislike copy/paste programming I have implemented a utility that helps to create such methods. I called this utility class ValueOf. It has 2 public methods: public static , V> T valueOf(Class enumType, String fieldName, V value); which finds the required field in specified enum. It is implemented utilizing reflection and uses a hash table initialized during the first call for better performance. The other overridden valueOf() looks like: public static > T valueOf(Class enumType, Comparable comparable); This method does not cache results, so it iterates over enum members on each invocation. But it is more universal: you can implement comparable as you want, so this method may find enum members using more complicated criteria. Full code with examples and JUnit test case are available here. Conclusions Java Enums provide the ability to locate enum members by name. This article describes a utility that makes it easy to locate enum members by any other field.

It's been almost 4 years since I wrote a blog post called "EasyMock - Pros, Cons, and Best Practices, and a lot has happened since. You don't hear about EasyMock much any more, and Mockito seems to have replaced it in mindshare. And for good reason: it is better. A Good Humane Interface for Stubbing Just like EasyMock, Mockito allows you to chain method calls together to produce less imperative looking code. Here's how you can make a Stub for the canonical Warehouse object: Warehouse mock = Mockito.mock(Warehouse.class); Mockito.when(mock.hasInventory(TALISKER, 50)). thenReturn(true); I know, I like a crazy formatting. Regardless, giving your System Under Test (SUT) indirect input couldn't be easier. There is no big advantage over EasyMock for stubbing behavior and passing a stub off to the SUT. Giving indirect input with mocks and then using standard JUnit asserts afterwards is simple with both tools, and both support the standard Hamcrest matchers. Class (not just Interface) Mocks Mockito allows you to mock out classes as well as interfaces. I know the EasyMock ClassExtensions allowed you to do this as well, but it is a little nicer to have it all in one package with Mockito. Supports Test Spies, not just Mocks There is a difference between spies and mocks. Stubs allow you to give indirect input to a test (the values are read but never written), Spies allow you to gather indirect output from a test (the mock is written to and verified, but does not give the test input), and Mocks are both (your object gives indirect input to your test through Stubbing and gathers indirect output through spying). The difference is illustrated between two code examples. In EasyMock, you only have mocks. You must set all input and output expectations before running the test, then verify afterwards. // arrange Warehouse mock = EasyMock.createMock(Warehouse.class); EasyMock.expect( mock.hasInventory(TALISKER, 50)). andReturn(true).once(); EasyMock.expect( mock.remove(TALISKER, 50)). andReturn(true).once(); EasyMock.replay(mock); //act Order order = new Order(TALISKER, 50); order.fill(warehouse); // assert EasyMock.verify(mock); That's a lot of code, and not all of it is needed. The arrange section is setting up a stub (the warehouse has inventory) and setting up a mock expectation (the remove method will be called later). The assertion in all this is actually the little verify() method at the end. The main point of this test is that remove() was called, but that information is buried in a nest of expectations. Mockito improves on this by throwing out both the record/playback mode and a generic verify() method. It is shorter and clearer this way: // arrange Warehouse mock = Mockito.mock(Warehouse.class); Mockito.when(mock.hasInventory(TALISKER, 50)). thenReturn(true); //act Order order = new Order(TALISKER, 50); order.fill(warehouse); // assert Mockito.verify(warehouse).remove(TALISKER, 50); The verify step with Mockito is spying on the results of the test, not recording and verifying. Less code and a clearer picture of what really is expected. Update: There is a separate Spy API you can use in Mockito as well: http://mockito.googlecode.com/svn/branches/1.8.3/javadoc/org/mockito/Mockito.html#13 Better Void Method Handling Mockito handles void methods better than EasyMock. The fluent API works fine with a void method, but in EasyMock there were some special methods you had to write. First, the Mockito code is fairly simple to read: // arrange Warehouse mock = Mockito.mock(Warehouse.class); //act Order order = new Order(TALISKER, 50); order.fill(warehouse); // assert Mockito.verify(warehouse).remove(TALISKER, 50); Here is the same in EasyMock. Not as good: // arrange Warehouse mock = EasyMock.createMock(Warehouse.class); mock.remove(TALISKER, 50); EasyMock.expectLastMethodCall().once(); EasyMock.replay(mock); //act Order order = new Order(TALISKER, 50); order.fill(warehouse); // assert EasyMock.verify(mock); Mock Object Organization Patterns Both Mockito and EasyMock suffer from difficult maintenance. What I said in my original EasyMock post holds true for Mockito: The method chaining style interface is easy to write, but I find it difficult to read. When a test other than the one I'm working on fails, it's often very difficult to determine what exactly is going on. I end up having to examine the production code and the test expectation code to diagnose the issue. Hand-rolled mock objects are much easier to diagnose when something breaks... This problem is especially nasty after refactoring expectation code to reduce duplication. For the life of me, I cannot follow expectation code that has been refactored into shared methods. Now, four years later, I have a solution that works well for me. With a little care you can make your mocks reusable, maintainable, and readable. This approach was battle tested over many months in an Enterprise Environment(tm). Create a private static method the first time you need a mock. Any important data needs to be passed in as a parameter. Using constants or "magic" fields hides important information and obfuscates tests. For example: User user = createMockUser("userID", "name"); ... assertEquals("userID", result.id()); assertEquals("name", result.name(); Everything important is visible and in the test, nothing important is hidden. You need to completely hide the replay state behind this factory method if you're still on EasyMock. The Mock framework in use is an implementation detail and try not to let it leak. Next, as your dependencies grow, be sure to always pass them in as factory method parameters. If you need a User and a Role object, then don't create one method that creates both mocks. One method instantiates one object, otherwise it is a parameter and compose your mock objects in the test method: User user = createMockUser( "userID", "name", createMockRole("role1"), createMockRole("role2") ); When each object type has a factory method, then it makes it much easier to compose the different types of objects together. Reuse. But you can only reuse the methods when they are simple and with few dependencies, otherwise they become too specific and difficult to understand. The first time you need to reuse one of these methods, then move the method to a utility class called "*Mocker", like UserMocker or RoleMocker. Follow a naming convention so that they are always easy to find. If you remembered to make the private factory methods static then moving them should be very simple. Your client code ends up looking like this, but you can use static imports to fix that: User user = UserMocker.createMockUser( "userID", "name", RoleMocker.createMockRole("role1"), RoleMocker.createMockRole("role2") ); User overloaded methods liberally. Don't create one giant method with every possible parameter in the parameter list. There are good reasons to avoid overloading in production, but this is test. Use overloading so that the test methods only display data relevant to that test and nothing more. Using Varargs can also help keep a clean test. Lastly, don't use constants. Constants hide the important information out of sight, at the top of the file where you can't see it or in a Mocker class. It's OK to use constants within the test case, but don't define constants in the Mockers, it just hides relevant information and makes the test harder to read later. Avoid Abstract Test Cases Managing mock objects within abstract test cases has been very difficult for me, especially when managing replay and record states. I've given up mixing mock objects and abstract TestCase objects. When something breaks it simply takes too long to diagnose. An alternative is to create custom assertion methods that can be reused. Beyond that, I've given up on Abstract TestCase objects anyway, on the grounds of preferring composition of inheritance. Don't Replace Asserts with Verify My original comments about EasyMock are still relevant for Mockito: The easiest methods to understand and test are methods that perform some sort of work. You run the method and then use asserts to make sure everything worked. In contrast, mock objects make it easy to test delegation, which is when some object other than the SUT is doing work. Delegation means the method's purpose is to produce a side-effect, not actually perform work. Side-effect code is sometimes needed, but often more difficult to understand and debug. In fact, some languages don't even allow it! If you're test code contains assert methods then you have a good test. If you're code doesn't contain asserts, and instead contains a long list of verify() calls, then you're relying on side effects. This is a unit-test bad smell, especially if there are several objects than need to be verified. Verifying several objects at the end of a unit test is like saying, "My test method needs to do several things: x, y, and z." The charter and responsibility of the method is no longer clear. This is a candidate for refactoring. No More All or Nothing Testing Mockito's verify() methods are much more flexible than EasyMock's. You can verify that only one or two methods on the mock were called, while EasyMock had just one coarse verify() method. With EasyMock I ended up littering the code with meaningless expectations, but not so in Mockito. This alone is reason enough to switch. Failure: Expected X received X For the most part, Mockito error messages are better than EasyMock's. However, you still sometimes see a failure that reads "Failure. Got X Expected X." Basically, this means that your toString() methods produce the same results but equals() does not. Every user who starts out gets confused by this message at some point. Be Warned. Don't Stop Handrolling Mocks Don't throw out hand-rolled mock objects. They have their place. Subclass and Override is a very useful technique for creating a testing seam, use it. Learn to Write an ArgumentMatcher Learn to write an ArgumentMatcher. There is a learning curve but it's over quickly. This post is long enough, so I won't give an example. That's it. See you again in 4 years when the next framework comes out! From http://hamletdarcy.blogspot.com/2010/09/mockito-pros-cons-and-best-practices.html

The Separated Interface pattern can often be used to provide hook points to client code, in the form of interfaces to implement or classes to extend with client code. The right implementation to use in a part of the system can then be chosen via configuration: the Factory or Dependency Injection container with the largest scope would process the configuration and execute conditionals only one time, and inject the right Plugin as a collaborator of a standard object. This pattern is a evolution of the Separated Interface one, where the implementor package is not even under your maintenance, but it is provided by some external developer that links his code to your work. Implementation In PHP the concept of compile time does not exist, apart from the just-in-time cached compilation of the scripts to operation codes, a phrase which you can peacefully ignore if you are not into caching. By the way, even if some checks are performed while loading and parsing the PHP code, PHP is by design a dynamic language where you can write nearly everything and it will not explode until executed. This design leaves open many possibilities for inserting plugins, but due to the lack of compile there is often a lack of a clean separation between code and configuration. For example, database credentials are embedded in PHP code more often than in other languages. Think now of a framework or a library: you cannot change the code but you must adapt or create a configuration to make it work. To implement a Plugin pattern, your application should strive towards the flexibility of a library: think of your production code as external and untouchable, and try to deploy a particular configuration to make it work and to modify a functionality. For example, extract it in a temporary working copy with svn checkout or git clone and hook in the necessary extensions. When you succeed, and your svn diff or git diff is clean, you'll have implemented a Plugin system. Modification of vendor code (and you are the vendor here) is out of the question. Future changes Kent Beck says in Implementation Patterns that providing hooks via implementation and inheritance is one of the most effective ways to tie a framework down from future evolution. For example, once you have published an interface, you cannot add methods to it without breaking all the implementors. You can publish versioned interfaces, but this adds complexity to your application. With a published abstract class instead, you can include a default implementation for new methods, but you can't remove methods or refactor protected members without breaking Plugin implementators. This is the specular situation of providing an interface. Zend Framework includes both an interface and an abstract class for most of its components, but it does not get right the management of extension points (at least in the 1.x branch). When including the possibility of Plugins in your application, default as much as possible to private visibility and hide the internals of your Plugin hook point. What is left to protected is a seam that screams "extend me", and the interfaces not marked as internal will be implemented by someone else. There is no built-in language mechanism to protect interfaces,m so you'll have to rely on some kind of convention (like a particular prefix or namespace), but for private methods left to protected scope we can only blame ourselves. Configuration The configuration of your Plugin system can be managed with solution of different levels of complexity, each more powerful than the previous ones. Of course, you shouldn't provide a needlessly complex system when all you need is a class name. The first solution is indeed to insert class names into configuration files. This is a totally declarative approach, which uses simple INI files. This is commonly done in Zend Framework, for example with bootstrap resources, and in some cases can even manage dependencies of the Plugins. Bootstrap resources can request other object of the same kind, but cannot pull in arbitrary collaborators (unless they create them by themselves... ugly if you know what DI is). A second, widely applicable solution is to request Factory objects. this solution still involves writing PHP code, but it is one step towards textual configuration. However, a Factory object can fetch and inject all the dependencies into a Plugin without cluttering it with this kind glue code (only a constructor or some setters). The problem with Factories is that they tend to contain all the same boilerplate code. A third solution can be used to provide quick construction of objects: Dependency Injection containers, which have recently been introduced even in PHP. A DI container is configured textually, via an XML or INI file containing parameters like the collaborators each object requires, its lifetime, and so on. DI containers are probably the future of flexible PHP applications, but beware of growing too dependent on them: they are a library like every other open source component, and should be isolated from your code as much as possible like you would do with your models and Doctrine 2, or your services and Zend Framework. Example The code sample shows hot to predispose a class for receiving an injected simple Factory that manages user-defined plugins. // plugin_view.php formatDate(time()), ".\n"; factory = $factory; } public function render($script) { include $script; } /** * Forwards the call to the View Helper invoked. */ public function __call($name, $args) { $callback = $this->factory->getHelper($name); return call_user_func_array($callback, $args); } } /** * Extension code. */ class UserDefinedFactory implements ViewHelperFactory { private $helpers; /** * In this example, we only define a simple Plugin for * formatting dates using PHP's internal function. */ public function __construct() { $this->helpers = array( 'formatDate' => function($time) { return date('Y-m-d', $time); } ); } public function getHelper($name) { return $this->helpers[$name]; } } // client code $view = new View(new UserDefinedFactory); $view->render('plugin_view.php');

The Special Case pattern is a very simple base pattern that describes a subclass representing, as the name suggests, a special case of the computation made by your program. Don't think that the technical simplicity of the solution means that this pattern is very diffused. If vs. polymorphism The idea of the pattern is to implements two classes with the same interface or base superclass, and rely on polymorphism to target the special case, instead that inserting if and switch statements in the original class. The extracted piece of functionality can be a method to override (specialization in the Template Method pattern) or an independent collaborator injected into the client code (Strategy pattern and many others). Dispatching a method call instead of inserting if statements is simpler to read and understand, as the code of the class has a lower cyclomatic complexity overall (few possible execution paths). If you ever tried to debug Doctrine 1 or a similar piece of software where the methods contain many nested ifs, you have been probably forced to insert echo statements to reveal the actually executed path, even when a single, isolated unit test was exercising the code. The alternative to some ifs is to introduce a Special Case. A rule of thumb for discovering if the substitution is possible is to check if the condition of the if is based on a state that is longer lived with respect to the parameters of the method that it resides in. Ifs that depend only on the state of the object fields or on collaborators are the simplest to replace. Null Object The Null Object pattern is a specialized version of Special Case (what a pun), and probably the most famous one. Instead of returning false or null when a computation fails to provide a result, you return an object that as a matter of fact, does nothing: an User subclass AnonymousUser with authorize() that always return false an empty array (it can be thought of as a Null Object, even if it is a primitive value) an empty ArrayObject. When a Null Object is returned, it effectively removes the checks for the null value or empty result from the client code. The client class object can call methods on the return value without worrying (calling methods on NULL is a fatal error in PHP and would crash a test suite). Or it can execute foreach() over a returned array and skip the cycle altogether if the array is empty. Since null can be dispatched, we may in fact use it as a Test Double in our test code to ensure a collaborator is never called in a particular scenario. If you have a method that shouldn't refer to a collaborator, you can inject null in the constructor or via a setter. PHP is different from Java in the type hinting behavior: in Java you can pass null to this constructor: public MyClass(Collaborator c) { ... while in PHP you have to resort to this: public function __construct(Collaborator $c = null) { ... Implementation The Special Case can be a Flyweight or an object with, since it has usually no internal state: the behavior depending on state is encapsulate in the code itself. You can also have more than one Special Case for each superclass or interface: Fowler makes the example of a MissingCustomer and AnonymousCustomer as special cases for the Customer class. By the way, every method of a Null Object should return a plain scalar value or another Special Case object. Note that with more than one level of Special Case objects, you may be violating the Law of Demeter: your client code access the first Special Case and then the other contained one, navigating the object graph instead of asking for its dependencies or sending a message. Examples In this example we apply the pattern to go from this situation: type = $type; } /** * This if() is based only on the object state * and can probably be modelled differently. * You'll need two tests for this method. */ public function accelerate() { if ($this->type == 'Ferrari') { $this->speed += 2; } else { $this->speed++; } } public function brake() { $this->speed--; } public function __toString() { return $this->type; } } // client code $car = new Car('Fiat'); $ferrari = new Car('Ferrari'); $car->accelerate(); $ferrari->accelerate(); var_dump($car, $ferrari); to this one: speed--; } public function __toString() { return $this->type; } } /** * One Special Case: a car with $type parametrized in the constructor * and ordinary acceleration properties. */ class OrdinaryCar extends Car { public function __construct($type) { $this->type = $type; } public function accelerate() { $this->speed++; } } /** * Another Special Case: a car with fixed $type and greater acceleration. */ class FerrariCar extends Car { /** * This is state encapsulate in code: you don't have to set up * it in tests or with configuration, only to instantiate this class. */ protected $type = 'Ferrari'; public function accelerate() { $this->speed += 2; } } // client code $car = new OrdinaryCar('Fiat'); $ferrari = new FerrariCar(); $car->accelerate(); $ferrari->accelerate(); var_dump($car, $ferrari);