Accurate Quantitative Analysis With ChatGPT and Azure AI Hub

LLMs are not very good at quantitative analysis. For example, when I asked ChatGPT, "Which number is bigger, 9.9 or 9.11?" it incorrectly responded with 9.11.

In another example, I have an Excel file containing a large amount of quantitative data. The maximum word count for a single prompt that ChatGPT can handle is around 4,000 words (approximately 16,000 characters). If I try to query this data, it may not fit within the context window. 

I can store the file in a data source and train ChatGPT to use that data source for user queries (check this article if you want to learn how to train ChatGPT on private data). However, since the file contains quantitative data, there is a chance that I might receive incorrect answers for complex queries.

How can such a problem be resolved? The answer likely involves a combination of data analysis using Python and LLMs.

Let's help ChatGPT provide the correct answer to “9.9 or 9.11, which number is bigger.”

A Step-by-Step Guide

Log in with https://ai.azure.com and click on Create project.

Give the Project a name; here, I named it LLM_Python, and clicked on Create.

Now, it is time to deploy an LLM like ChatGPT. Click on Models + endpoints.

Click on Deploy base model.

Select the LLM of your choice. I have selected gtp-4o-mini. Next, click on Confirm.

Click on Deploy. This step completes the deployment of LLM, which is gpt-4o-min in this example.

Click on Prompt flow.

Click on + Create.

Click on Create on Standard flow.

Wait a couple of minutes because the new flow takes some time to propagate. Give a Folder name (I have given it codefolder) and then click on Create.

The prompt is ready now. To customize the programming, click on the delete button under the red circle.

Click on + Add input.

I selected Question for Input name and 9.9 or 9.11 which number is bigger? for the Value. Basically, this is a user question to chat-4o-mini (LLM).

Now click on + LLM and provide the node name, I gave node name FirstLLMCall and clicked on Add.

Select the highlighted values as per your project and provide instructions to the LLM for what you want it to do. I want the LLM to extract two numbers from the user input. Below are the instructions for gpt-40-mini (LLM):

#system:

You are a helpful assistant.

#user:

Extract the two numbers from the following question: "{{user_query}}".

Return only a valid JSON object without any extra formatting or explanations.

Ensure it is properly structured like this:

{
  "num1": 19.9,
  "num2": 19.11
}

Click on Start compute session. It will take around 10 minutes to complete.

Once Start compute session changes to Compute session running, click on Validate and parse input.

Select ${inputs.Question} from Value box. It stores user input and passes it to LLM.

Click on + Python to write code to get greater value from user-provided numbers.

Provide a node name for Python; I gave it PythonCode. Next, click on Add.

Copy and paste the below Python code in the PythonCode prompt:

import json
from promptflow import tool
@tool
def find_max_number(input_json: str) -> str:
    """
    This function takes a JSON string with two numbers,
    finds the maximum, and returns the result as a JSON string.
    """
    # Read extracted numbers from the previous ChatGPT output
    input_data = json.loads(input_json)
    num1 = float(input_data["num1"])
    num2 = float(input_data["num2"])

    # Find the maximum number
    max_number = max(num1, num2)

    # Return the result in JSON format
    output_json = json.dumps({"num1": num1, "num2": num2, "max_number": max_number})
    return output_json

Click on Validate and parse input.

Now, select $FirstLLMCall.output in Value.

We need another instance of an LLM to provide answers to user questions based on the output of the above Python code. Click on LLM to proceed.

I gave the node the name AnsweringLLM and clicked on Add.

Select values for AnsweringLLM.

Write code on AnsweringLLM and click on Validate and parse input.

#system:

You are a helpful assistant.

# user:

The numbers extracted are {{num1}} and {{num2}}. The bigger number is {{max_number}}.

Give your answer in 1 line only.

Select inputs for the AnsweringLLM.

Click on + Add output, as the user will be provided output from AnsweringLLM.

In the Answer value, select ${AnsweringLLM.output}.

Below is the flow of requests:

Click on Save and then Run.

Now, click on View outputs to check what was the answer to the user question.

We got the correct answers this time.

Summary

When we asked the question, ‘9.9 or 9.11, which number is bigger?’ The answer was 9.11. This indicates that ChatGPT was unable to provide the correct answer for quantitative data, but with the help of Python, ChatGPT provided the correct answer, so in such a situation, it is better to use Python with LLM.