DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. AI/ML
  4. Intelligent Agents: Machine Reading Comprehension

Intelligent Agents: Machine Reading Comprehension

Read on in order to discover more information on AI models and machine reading comprehension. Are AI models surpassing humans?

Somnath Biswas user avatar by
Somnath Biswas
·
May. 18, 18 · News
Like (1)
Save
Tweet
Share
8.14K Views

Join the DZone community and get the full member experience.

Join For Free

On 5th of January this year, an AI model, for the first time, outperformed humans in reading comprehension. The SLQA+ (ensemble) model from Alibaba recorded an exact match score of 82.44 against the human score of 82.304, on the SQuAD dataset.

It turns out that the Microsoft r-net+ (ensemble) model had achieved 82.650 two days prior to that. And since then, two other models have also gone on to beat the human EM score. While none of the models have beaten the human F1 Score (precision, recall) of 91.21 yet, these events further underline the frantic pace at which the RC models are evolving, which is great news because Reading Comprehension (RC) is a key element of intelligent agent systems.

Intelligent Agents & Machine Reading Comprehension

Building intelligent agents with the ability to answer open-domain and even closed-domain questions with high accuracy has been a key goal of most AI labs. Intelligent agents with RC and Question-Answer (QA) abilities can help AI personal assistant systems like Alexa, Google Assistant, Siri, Cortana, etc. perform better and help enterprises in having intelligent agent bots supplement human agents or directly process chat & messaging traffic and maybe even voice to some extent.

Machine Comprehension/Machine Reading Comprehension/Machine Reading models enable computers to read a document and answer general questions against it. While this is a relatively elementary task for a human, it's not that straightforward for AI models. There are multiple NTM (Neural Turning Machine), Memory Network, and Attention Models for Reading Comprehension available. The list of SQuAD models can be accessed here.

As a first step towards building our Intelligent agent system (humanly.ai), we are also building a machine reading system. Our implementation is based on the BiDAF (Bi-Directional Attention Flow) ensemble model & Textual Entailment. It's still work in progress (EM 67%, and F1 77%), and sometimes it gives funny answers but you can try it out here.

Key Challenges

One of the basic challenges we faced was handling the questions that would require a yes/no type of answer (a further inference between the question, answer, and the document) — and hence the implementation of the Textual Entailment module. The other observation was to respond back in full sentences ("Yes, Narendra Modi is the Prime Minister of India" instead of a "Yes", to the question, "Is Narendra Modi the Prime Minister of India?"), and for that as the next product increment we are currently planning on implementing the Seq2Seq model to format our responses.

But one major challenge, all machine reading systems face, especially when it comes to practical implementations for specific domains or verticals, is the absence of supervised learning data (labeled data) for that domain. All the contemporary, reading comprehensions models are built on supervised training data with labeled questions and answers, a paragraph with the answer, etc. So when it comes to new domains, whilst the Enterprises have artifacts and data, the absence of labeled data presents a challenge.

We are currently experimenting with an ensemble of machine reading comprehension models, each trained on a specific dataset, so that the learning is incremental. While the scores are improving for the model, the need for labeled domain data to train the MRC model in the first place still persists. Towards this problem, I came across two very neat solutions, which attempt domain transference from Microsoft — SynNet and ReasoNet, which we intend to explore further.

The 'two-stage Synthesis Networks' or SynNet model first gets trained on supervised data for a given vertical and learns the technique to identify patterns for critical information (named entities, knowledge points etc.) and then generates questions around these answers. Once trained, it can then generate pseudo-questions and answers against artifacts for the new domain. These can then be used to train the MRC on the new domain.

The Reasoning Network, or ReasoNet, essentially uses reinforcement learning to dynamically figure when it has enough information to answer a question, and that it should stop reading. This is a deviation from the approach of using a fixed number of turns during the process of inferring the relationship between the questions, artifacts, and the answers. This has also performed exceptionally when on the SQuAD dataset.

We Shall Overcome

As various models continue to emerge, its a reasonable guess that sooner rather than later (especially catalyzed by the availability of so many datasets that are themselves growing rapidly) that machine comprehension models will be able to overcome the key challenges and get us closer to the goal of intelligent agents that can be trained on standard documents and answer general questions — as humans do.

I do hope you found the post useful in getting some basic understanding of machine comprehension. As always, do leave your comments & thoughts — including any aspects that I might have missed. I will be more than happy to incorporate them.

Disclaimers: The above post in no way claims any copyright to any of the images or literature presented.

Machine learning

Published at DZone with permission of Somnath Biswas. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Cloud Native London Meetup: 3 Pitfalls Everyone Should Avoid With Cloud Data
  • DevSecOps Benefits and Challenges
  • Web Application Architecture: The Latest Guide
  • AIOps Being Powered by Robotic Data Automation

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: