How To Handle 100k Rows Decision Table in Drools (Part 3)
In this article, I created a prototype to demonstrate how to handle large rows in a decision table with reasonable performance. This is part 3 of the series.
Join the DZone community and get the full member experience.Join For Free
As described in the previous article, we are facing a very challenging performance issue when solving 100k row decision tables.
To view the previous discussion, please click:
Solution 1: Rule Template + XLS
Solution 2: Precompile Spreadsheet Decision Table
In this article, I am going to change the mindset and change the decision table row data from rule to fact. Again, I am using Drools as my framework.
Solution 3: Convert Row Data to Fact, Not Rule
Solution 2 (Precompile the rules) helps a lot when the decision table is 10k below row size. However, when the row number comes to very big such as 100k, solution 2's improvement is neglectable. What’s worse is that the compilation time is too long, this is because it’s not a normal task to compile 100k Java files in a project.
By its design nature, the decision table is popular since it allows you to manage one rule in one row so that you can manage multiple rows in an easy manner. However, the watch that floats the boat also could swallow it.
Could we somehow reduce the rule numbers if the rule numbers are just too big?
Could we read the Excel file in RAW format, insert those row data as Facts and then compute them in memory? Then you have 100k Facts instead of 100k rules in memory.
Let's try this:
Insert Row as Fact in Java
With a simple test on the above code, it works, and the performance is <50 ms!
However, we don’t throw away the bathwater with the child. We don’t want to hardcode the Excel Read and Parser in our generic application code. The business logic needs to be decoupled with a generic application.
Could we load Excel data in a DRL file and insert the fact?
The answer is yes!
DRL is very dynamic and flexible, basically, you can do it just like you are writing Java code.
However, in order to simplify the DRL editing, usually, we could provide a clean utility to read Excel files and convert them to a Customized Fact to serve your purpose.
I also provided a simple Excel utility in my branch, See KeywordReader.java, we can add the Excel parse in the Drools Model Getter and Setter.
How to parse Excel file:
Let’s run these rules in the client application:
Great, the performance is very good.
As you can see, loading the excel file in rule might take 2.7 seconds. After that, each rule execution only costs 70ms.
The biggest advantage is good performance. As you can see the performance is over 100 times faster than previous solutions considering the large number of input data in Excel files;
What’s even better, there is no compilation overhead.
The excel data can still be decoupled from application logic. All business rules data can be packaged in the rules project as usual.
There are two shortcomings as far as I can see:
- Raw Excel data can’t be handled in kie-workbench.
- There is some complex Excel Reader logic in the rule project.
It might not be comfortable for business users to maintain the Excel parser login in the Drools model, however, I think, you can wrap the Excel Parse logic in a separate jar file which would decouple it from a business rule project.
Opinions expressed by DZone contributors are their own.