Building a Better ERP Architecture to Support Machine Learning

Users and developers agree on one thing in relation to enterprise resource planning (ERP) software: it's boring.

For users, ERP means a lot of data entry added to their daily workload. While ERP is becoming more sophisticated with support for mobile devices and cloud functionality now commonplace, most users would prefer to use systems that are quicker and more responsive.

Developers can see this is a pain point and are trying to innovate past it, but that's not always possible with the kind of monolithic architecture that can be found in contemporary ERP installations. ERP is a challenge to developers because ERP systems communicate with so many other systems on the network and each ERP installation is so massively configurable. Each individual instance of an ERP can feel like a unique piece of software in itself.

Chatbots like Unit4’s Wanda are a good example of the kind of innovation that developers are creating in the ERP space. A chatbot-style interface allows users to perform actions such as adding new records to the ERP using natural language from the apps they use every day. It completely changes the ERP user experience, with some users interacting with the system but never having to access the ERP solution itself.

But to be of value, the digital assistant needs to have a little magic in the background that reduces the amount of work required from the user. If a chatbot simply asks the user to type in the value for each required field for example, then it is simply another way of filling out forms.

This is where machine learning comes in. DIMPS, or Dimensional Prophecy Service, can fill out the forms on the user's behalf by looking across historical data on a number of applications. The user experience is vastly improved by DIMPS, providing maximum value from the system with the minimum of user effort required.

The only problem? Where to get the data to make DIMPS possible.

Rethinking ERP Architecture

Because ERP systems have so many interdependencies on other systems, a project like DIMPS runs into several problems. Each system has its own data model and its own API to serve data back to the ERP, making it hard to ensure that data is in a usable format. Many systems also only write the final transaction data and don't store historical user data, which is crucial for DIMPS.

So, the first challenge is unifying all available data sources. An obvious solution is to build a data lake using a technology such as Microsoft Azure, but then there is the problem of ensuring that data coming into the lake is in a usable format, especially when coming from client systems that can't be directly controlled.

The first step then is to build an ontology, or canonical data model, which is a unified data format that can be used across diverse data sources. JSON-based Enterprise Documents capture the dimensions that are needed for further analysis and ensure everything in the data lake is in a usable format.

What about legacy systems though? Some vital systems are too antiquated to configure with the canonical data model, and developers have to find ways to extract the data from the system so they can pass it to the lake.

To deal with this problem, companies can use a model that embraces the ERP system in a service layer composed of whatever technology is compatible with the system in order to extract the required data, whether that's using SOAP calls or querying the database directly. The system can run as normal but with the service layer generating the required enterprise documents.

These are some of the techniques to get towards the end goal, which is a data lake rich enough to power machine learning, enabling DIMPS to create a better user experience. But one of the key issues is speed. How does one capture data in real-time to allow instantaneous processing and a seamless user experience?

A good approach can be to make changes within the data access layer in the existing architecture. This will now become the event processing layer, performing the database action while also sending a message to the microservice, creating the enterprise document. Here, the system can also build up the historical user data that is required to anticipate user choices — the prophecy part of DIMPS.

This new architecture will not only provide more data than ever before, it can also give a better quality of data in a unified and predictable format. From here, rich analytics can help to build sophisticated UIs and facilitate the machine learning and artificial intelligence that will be at the heart of all enterprise software from now on. It also lays the groundwork for future innovation, which is why ERP is actually one of the more exciting fields for developers right now.

When this model is employed, users will no longer find themselves opening their ERP client and having to fill in a spreadsheet-style form, such as a timesheet. Instead, they will start a chat session with any chatbot and issue a natural language command, such as "submit a timesheet for last week." The chatbot will use DIMPS to deduce all of the data required to complete the form, and the user will only have to approve.

All of the organizational benefit of ERP with little of the administrative overhead. Machine learning is a big win for everyone.