← Writing

Introducing PandasAI

·4 min read·AI

I built PandasAI because I was sick of writing the same groupby-filter-plot boilerplate for questions that should take one sentence to ask.

What it is

PandasAI wraps a pandas dataframe with a language model. You ask a question in plain English, it writes the pandas code, runs it, and gives you the answer. No prompt engineering on your end.

import pandas as pd
from pandasai import PandasAI
from pandasai.llm.openai import OpenAI

df = pd.DataFrame({
    "country": ["United States", "United Kingdom", "France", "Germany", "Italy"],
    "gdp": [19294482071552, 2891615567872, 2411255037952, 3435817336832, 1745433788416],
    "happiness_index": [6.94, 7.16, 6.66, 7.07, 6.38],
})

llm = OpenAI(api_token="YOUR_API_TOKEN")
pandas_ai = PandasAI(llm)
pandas_ai.run(df, prompt="Which are the 5 happiest countries?")

It returns a dataframe, a number or a sentence, depending on the question. Ask for a chart and you get a matplotlib plot.

How it works

PandasAI doesn't send your data to the model. It sends the question, the column names and the first few rows, enough for the model to understand the shape of the dataframe. The model writes Python against that shape, and PandasAI runs the code locally, on the full dataframe, on your machine. If even the first rows are sensitive, there's a mode that sends only the column names.

Diagram: the question, column names and first few rows go from your machine to the language model. The model sends back pandas code, which runs locally on the full dataframe and returns a number, a table or a chart.
Only the shape of the data leaves your machine. The code comes back and runs where the rows are.

That design came from a practical constraint more than a principle. Nobody is going to paste a customer table into a chatbot, and they're right not to. Keeping the data where it already lives, and moving only the question and the code, is the version of this that people can use at work.

It runs on OpenAI's models by default, and you can swap in open models from Hugging Face, like StarCoder, if you'd rather not depend on a single provider. That list will change every few weeks the way things are moving.

Why now

Two months ago I wouldn't have bothered. GPT-4 changed my mind. Earlier models wrote pandas code that looked right and failed often enough that you had to check every line. GPT-4 gets ordinary analysis questions right often enough that checking its work is faster than writing the code yourself. That's the threshold that matters for a tool like this.

OpenAI is clearly heading the same way with Code Interpreter for ChatGPT. I take that as good news, since it shows the idea works, but it means uploading your files to OpenAI and working inside their chat window. PandasAI goes the other way. It lives in the notebook, the script or the pipeline where your dataframe already is, and it's open source, so you can read every line of what it does.

Why it matters beyond saving me ten lines

Every data team has the same problem. The people who need an answer from the data can't write the code, and the people who can write the code are busy. So the questions queue up, the answers arrive a week late, and most questions never get asked at all because asking costs too much.

For decades the interface to data has been a query language: SQL, then pandas, then whatever BI tool sits on top and gives you a fixed set of charts. Each one made data more accessible, and each one still asked people to learn a language before they could ask a question. Language models remove that step. The question itself becomes the query.

Four ways to ask for the five happiest countries. SQL in the 1970s: ORDER BY happiness_index DESC LIMIT 5. Pandas in 2008: df.nlargest(5, 'happiness_index'). BI dashboards in the 2010s: the chart someone built for you. In 2023: the question itself, with nothing new to learn.
Each interface asked a little less of the person with the question.

I think that's where this goes. Today PandasAI is a shortcut for people who already know pandas, and that's a fine place to start. Next it needs to handle several dataframes at once, remember the conversation so you can ask a follow-up, connect straight to a SQL database instead of a CSV, and explain what it did so a non-technical user can trust the answer. Eventually the person asking the question isn't an analyst at all. It's the product manager, the salesperson or the founder who has had to wait in that queue.

Same bet I made with ChatPic, different domain: the interface should be the thing you'd say out loud.

What it isn't

It's not magic, and it won't save you from a messy dataset. It gets things wrong sometimes, which is true of every LLM-backed tool, and I'd rather say that upfront than pretend otherwise. A wrong answer that looks confident is worse than no answer, so making the generated code easy to inspect is near the top of the list.

There's also the obvious risk: PandasAI runs code a model wrote. I'm adding guardrails around what that code is allowed to import and call, but for now treat it as a tool for exploring data on your own machine, not something to point at production.

Where it stands

The code is open source: PandasAI repo. The response over the past few weeks has been far bigger than I expected, and the issues people open are already steering the roadmap. It's rough in places. That's what early users and bug reports are for: file one if you hit something, or send a PR if you fix it yourself.