How To Handle 100k Rows Decision Table in Drools (Part 3)

In this article, I created a prototype to demonstrate how to handle large rows in a decision table with reasonable performance. This is part 3 of the series.

· Performance Zone · Analysis
Save
3.41K Views

As described in the previous article, we are facing a very challenging performance issue when solving 100k row decision tables.

To view the previous discussion, please click:

Solution 1: Rule Template + XLS

Solution 2: Precompile Spreadsheet Decision Table

In this article, I am going to change the mindset and change the decision table row data from rule to fact. Again, I am using Drools as my framework.

Solution 3: Convert Row Data to Fact, Not Rule

Solution 2 (Precompile the rules) helps a lot when the decision table is 10k below row size. However, when the row number comes to very big such as 100k, solution 2's improvement is neglectable. What’s worse is that the compilation time is too long, this is because it’s not a normal task to compile 100k Java files in a project.

By its design nature, the decision table is popular since it allows you to manage one rule in one row so that you can manage multiple rows in an easy manner. However, the watch that floats the boat also could swallow it.

Could we somehow reduce the rule numbers if the rule numbers are just too big?

Could we read the Excel file in RAW format, insert those row data as Facts and then compute them in memory? Then you have 100k Facts instead of 100k rules in memory.

Let's try this:

DRL

Java

x

1
`rule "Check Client Object"`
2
`    when`
3
`        \$k:Keyword(\$v : value)`
4
`        \$c:ClientObject( descr matches \$v )`
5
`    then`
6
`        \$c.setPass(\$k.isResult());`
7
`        System.out.println("Check Client Object fired");`
8
`end`

Drools Model

Java

`xxxxxxxxxx`
1

1
`public class Keyword {`
2
`    private java.lang.String value;`
3

4
`    private boolean result;`
5
`}`

Insert Row as Fact in Java

Java

`xxxxxxxxxx`
1

1
`ClientObject o1 = new ClientObject();`
2
`for(int i=0;i<100000;i++){`
3
`  ksession.insert(new Keyword("DangerObjectxxx", false);`
4
`}`
5
`o1.setDescr("999");`
6
`int fired = ksession.fireAllRules();`

With a simple test on the above code, it works, and the performance is <50 ms!

However, we don’t throw away the bathwater with the child. We don’t want to hardcode the Excel Read and Parser in our generic application code. The business logic needs to be decoupled with a generic application.

Could we load Excel data in a DRL file and insert the fact?

The answer is yes!

DRL is very dynamic and flexible, basically, you can do it just like you are writing Java code.

However, in order to simplify the DRL editing, usually, we could provide a clean utility to read Excel files and convert them to a Customized Fact to serve your purpose.

Improved DRL

Java

`xxxxxxxxxx`
1
20

1
`rule "Load keyword"`
2
`    when`
3
`        \$kr:KeywordReader(excelFile matches ".*.xls")`
4
`    then`
5
`        System.out.println("Load keyword rule fired");`
6
`        List<Map<String, String>> list = \$kr.getKwList();`
7
`        for ( Map<String, String> m : list){`
8
`            Keyword k = new Keyword(m.get("1"), Boolean.valueOf(m.get("2")));`
9
`            insert(k);`
10
`        }`
11
`end`
12

13
`rule "Check Client Object"`
14
`    when`
15
`        \$k:Keyword(\$v : value)`
16
`        \$c:ClientObject( descr matches \$v )`
17
`    then`
18
`        \$c.setPass(\$k.isResult());`
19
`        System.out.println("Check Client Object fired");`
20
`end`

I also provided a simple Excel utility in my branch, See KeywordReader.java, we can add the Excel parse in the Drools Model Getter and Setter.

Java

`xxxxxxxxxx`
1
22

1
`public class KeywordReader {`
2
`    private String excelFile;`
3
`    private List<Map<String, String>> kwList;`
4

5
`    public KeywordReader(String excelFile) {`
6
`        this.excelFile = excelFile;`
7
`        kwList= new ArrayList<Map<String, String>>();`
8
`    }`
9

10
`    public String getExcelFile() {`
11
`        return excelFile;`
12
`    }`
13

14
`    public List<Map<String,String>> getKwList() {`
15
`        if(this.excelFile.isEmpty())return null;`
16
`        parseFile(getClass().getClassLoader().getResourceAsStream(excelFile));`
17
`        return kwList;`
18
`    }`
19
`    `
20
`    private void parseFile( InputStream inStream ) {`
21
`        //Parser Excel format and conver them into HashMapList`
22
`    }`

How to parse Excel file:

Java

`xxxxxxxxxx`
1
22

1
`Sheet sheet = workbook.getSheetAt( 0 );`
2
`int maxRows = sheet.getLastRowNum();`
3
`DataFormatter formatter = new DataFormatter( Locale.ENGLISH );`
4
`FormulaEvaluator formulaEvaluator = sheet.getWorkbook().getCreationHelper().createFormulaEvaluator();`
5
`        //define start row in rowFrom variable`
6
`        for ( int i = rowFrom; i <= maxRows; i++ ) {`
7
`            Row row = sheet.getRow( i );`
8
`            int lastCellNum = row != null ? row.getLastCellNum() : 0;`
9
`            `
10
`            Map<String, String> rowData= new HashMap<String, String>();`
11
`            int index =0;`
12
`          //define start column in colFrom variable`
13
`            for ( int cellNum = colFrom; cellNum < lastCellNum; cellNum++ ) {`
14
`                Cell cell = row.getCell( cellNum );`
15
`                if ( cell == null ) {`
16
`                    continue;`
17
`                }`
18
`                double num = 0;`
19
`                index++;`
20
`                rowData.put(String.valueOf(index), cell.getStringCellValue());`
21
`         }`
22
`            kwList.add(rowData);`

Client Code

Java

`xxxxxxxxxx`
1

1
`ClientObject o1 = new ClientObject();`
2
`KeywordReader kr = new KeywordReader("100kTable.xls");`
3
`o1.setDescr("9999");`
4
`ksession.insert(o1);`
5
`ksession.insert(kr);`
6
`int fired = ksession.fireAllRules();`

Let’s run these rules in the client application:

Shell

`xxxxxxxxxx`
1

1
`mvn clean compile exec:java `
2
`Initial Kie Session elapsed time: 4943 `
3
`Load keyword rule fired`
4
`fired rules: 1 elapsed time: 2761 `
5
`Is Object Pass:true`
6
`Check Client Object fired`
7
`2second round fired rules: 1 elapsed time: 70 `

Great, the performance is very good.
As you can see, loading the excel file in rule might take 2.7 seconds. After that, each rule execution only costs 70ms.

Pros

The biggest advantage is good performance. As you can see the performance is over 100 times faster than previous solutions considering the large number of input data in Excel files;

What’s even better, there is no compilation overhead.

The excel data can still be decoupled from application logic. All business rules data can be packaged in the rules project as usual.

Cons

There are two shortcomings as far as I can see:

1. Raw Excel data can’t be handled in kie-workbench.
2. There is some complex Excel Reader logic in the rule project.

It might not be comfortable for business users to maintain the Excel parser login in the Drools model, however, I think, you can wrap the Excel Parse logic in a separate jar file which would decouple it from a business rule project.

Topics:
drools, drools business rules engine, decision table modeling, performance

Opinions expressed by DZone contributors are their own.