DataSage: The Proof-of-Concept That Took on a Life of it's Own

Data Intelligence

The “Aha!” Moment That Started This Journey

You know, I’ve spent my career believing that technology should empower everyone, not just the experts. That’s why, when I started working with data analytics at Stanford, I kept thinking about my days at Apple Computer, trying to understand a bill of materials in an Excel spreadsheet. Margins up, expenses down—but what did it all mean? What was the story those numbers were trying to tell?

That’s when it hit me: we’ve gotten really good at showing data, but we’re still terrible at explaining it. We’ve got these beautiful charts and graphs, sure, but they’re like giving someone sheet music when they just want to hear the song. And that’s where this whole adventure began.

So I built something. Not a finished product—more like a proof of concept, the way we used to breadboard circuits back in the day. I wanted to see if we could take traditional business intelligence tools and marry them to artificial intelligence in a way that would create actual narratives from data. Real stories that your spouce, your boss, or your Board of Directors could understand immediately.

The result? Stanford DataSage—a Stanford-branded data analysis platform that doesn’t just show you charts. It tells you what they mean.

The Problem: Data Without Context is Just Noise

Let me paint you a picture. Imagine you’re looking at 44 years of NFL division performance data. That’s eight divisions, each with win totals for every year since 1980. Now, if I hand you a spreadsheet with 352 data points (8 divisions × 44 years), your eyes are going to glaze over faster than a Krispy Kreme donut.

Even if I give you a nice line chart—let’s say I use Matplotlib, make it professional-looking with good colors—you’re still stuck doing the analysis yourself. You’ve got to spot the trends, identify the outliers, figure out what the 2008 dip means, compare divisions, and somehow turn all that into actionable intelligence.

That’s a lot of work. And here’s the thing: most people won’t do it. They’ll glance at the chart, say “interesting,” and move on. All that data, all those insights, wasted because the barrier to understanding was just too high.

This is exactly the kind of problem that gets me excited, because it’s solvable. We’ve got the technology now—we just need to wire it together the right way.

The Solution: Let AI Tell the Story

Here’s the core idea: what if the computer could look at your data, choose the best way to visualize it, create that visualization, and then explain it to you in plain English? Not just “Sales went up 15%,” but a real executive summary that synthesizes the patterns, explains what’s significant, and recommends what you should do about it.

That’s exactly what Stanford DataSage does. And the beautiful thing? The heavy lifting is done by Claude AI from Anthropic, but the orchestration—the conducting of this symphony—happens in a simple Flask application that anyone with basic Python knowledge could understand.

Let me show you how the magic happens.

The Architecture: Three Acts of Intelligence

Think of this system like a play in three acts:

Act One: The Analysis
Claude looks at your data structure and decides what kind of chart will tell the story best.

Act Two: The Visualization
Python and Matplotlib create that chart, using colorblind-friendly colors so everyone can see the story.

Act Three: The Narration
Claude analyzes the finished chart and writes an executive summary—complete with key findings, detailed analysis, and recommendations.

Let’s dig into each act.

Act One: Intelligent Chart Selection

This was the first “aha!” moment in building this system. Most data viz tools make you choose the chart type. Bar chart? Line chart? Scatter plot? But here’s the thing—if you already knew enough about your data to pick the right visualization, you probably wouldn’t need the tool in the first place!

So I flipped it around. Let the AI look at your data and pick the chart type for you.

Here’s how it works under the hood:

def get_chart_recommendation(df):
    """Ask Claude to recommend the best chart type for the data"""
    client = anthropic.Anthropic(api_key=ANTHROPIC_API_KEY)
    
    # Gather intelligence about the dataset
    num_rows = len(df)
    num_cols = len(df.columns)
    
    # Figure out which columns are labels vs. data
    id_columns = []
    for col in df.columns:
        try:
            pd.to_numeric(df[col], errors='raise')
            break  # Found the first data column
        except (ValueError, TypeError):
            id_columns.append(col)  # This is a label
    
    data_columns = df.columns[len(id_columns):].tolist()
    data_preview = df.head(5).to_string()
    
    # Ask Claude: "What's the best way to show this?"
    message = client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=500,
        messages=[{
            "role": "user",
            "content": f"""Analyze this dataset and recommend the 
            SINGLE best chart type.

            Dataset Info:
            - {num_rows} series/categories
            - {len(data_columns)} time periods/data points
            - Identifier columns: {', '.join(id_columns)}
            
            Data Preview:
            {data_preview}
            
            Choose ONE: LINE, BAR, SCATTER, or AREA
            
            Respond with ONLY the chart type and a brief reason.
            Format: "CHARTTYPE: reason"
            """
        }]
    )
    
    # Parse Claude's response
    response_text = message.content[0].text.strip()
    if 'LINE' in response_text.upper():
        return 'line', response_text
    elif 'BAR' in response_text.upper():
        return 'bar', response_text
    # ... and so on

What’s happening here is beautiful in its simplicity. We’re not trying to build a complex decision tree with rules like “if rows > 10 and columns < 20, use line chart.” Instead, we’re leveraging Claude’s training on thousands of data visualization examples. Claude has seen what works and what doesn’t. It can make nuanced judgments that would take us months to encode as rules.

When you upload that NFL data, Claude looks at it and says something like: “LINE: Time series data showing trends across multiple years is best visualized with line charts.” It’s not just picking randomly—it’s reasoning about your specific data.

Act Two: Dynamic Visualization

Now here’s where it gets fun. Once Claude has recommended a chart type, we need to actually create it. But remember—this has to work for any CSV file someone uploads. Could be car sales, could be NFL stats, could be your startup’s monthly revenue.

This is where good old-fashioned engineering comes in:

def generate_chart(df, chart_type='line'):
    """Generate a chart based on Claude's recommendation"""
    
    # Colorblind-friendly palette - because accessibility matters
    colors = [
        '#0173B2',  # Bright Blue
        '#DE8F05',  # Vibrant Orange
        '#029E73',  # Teal/Green
        '#CC78BC',  # Pink/Magenta
        # ... 4 more colors
    ]
    
    fig, ax = plt.subplots(figsize=(14, 8))
    
    # Smart column detection - works with any CSV structure
    id_columns = []
    for col in df.columns:
        try:
            pd.to_numeric(df[col], errors='raise')
            break
        except (ValueError, TypeError):
            id_columns.append(col)
    
    year_columns = df.columns[len(id_columns):].tolist()
    year_labels = [str(col) for col in year_columns]
    
    # Plot each row with the appropriate chart type
    for idx, row in df.iterrows():
        label = ' '.join([str(row[col]) for col in id_columns])
        values = pd.to_numeric(row[year_columns], errors='coerce').values
        color = colors[idx % len(colors)]
        
        if chart_type == 'line':
            ax.plot(year_labels, values, marker='o', label=label,
                   linewidth=3, color=color, markersize=5)
        
        elif chart_type == 'bar':
            x_positions = range(len(year_labels))
            width = 0.8 / len(df)
            offset = (idx - len(df)/2) * width + width/2
            ax.bar([x + offset for x in x_positions], values, 
                   width, label=label, color=color)
        
        elif chart_type == 'scatter':
            x_numeric = range(len(year_labels))
            ax.scatter(x_numeric, values, label=label,
                      color=color, s=100)
        
        elif chart_type == 'area':
            ax.fill_between(range(len(year_labels)), values,
                           label=label, color=color, alpha=0.6)
    
    # Make it pretty and professional
    ax.set_xlabel('Time Period', fontsize=13, fontweight='bold')
    ax.set_ylabel('Values', fontsize=13, fontweight='bold')
    ax.set_title(f'Data Trends - {chart_type.title()} Chart',
                fontsize=16, color='#8C1515')  # Stanford Cardinal Red
    ax.legend(loc='best')
    ax.grid(True, alpha=0.2)
    
    # Convert to base64 so we can send it over the web
    buffer = BytesIO()
    plt.savefig(buffer, format='png', dpi=150)
    buffer.seek(0)
    image_base64 = base64.b64encode(buffer.read()).decode()
    plt.close()
    
    return image_base64

The beauty of this code is in what it doesn’t assume. We don’t hard-code “column 1 is Make, column 2 is Model.” Instead, we dynamically detect where the labels end and the data begins. This means whether you upload a simple two-column dataset or a complex multi-identifier dataset like the NFL data, it just works.

And those colors? They’re not random. They’re a carefully selected palette that works for people with various types of color blindness. About 8% of men have some form of color vision deficiency—that’s 1 in 12 people. If we’re going to democratize data analysis, we need to make sure everyone can actually see the data.

Act Three: The Executive Summary

Now we get to the really cool part. We’ve got a beautiful chart. But a chart is still just a picture. We need words. We need context. We need a story.

This is where Claude truly shines. We send the chart image (as base64 encoded data) along with some statistics about the dataset, and we ask Claude to write an executive summary. But not just any summary—a structured, comprehensive analysis that could go straight into a board presentation.

Here’s the prompt structure:

def get_claude_analysis(chart_base64, df, chart_type, recommendation_reason):
    """Get a comprehensive executive analysis from Claude"""
    client = anthropic.Anthropic(api_key=ANTHROPIC_API_KEY)
    
    # Calculate summary statistics
    data_summary = "Data Summary:\n"
    for idx, row in df.iterrows():
        label = ' '.join([str(row[col]) for col in id_columns])
        numeric_values = pd.to_numeric(row[year_columns], errors='coerce')
        total = numeric_values.sum()
        average = numeric_values.mean()
        data_summary += f"- {label}: Total={total:,.0f}, Avg={average:,.0f}\n"
    
    message = client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=3500,  # Room for a detailed analysis
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": chart_base64,
                    },
                },
                {
                    "type": "text",
                    "text": f"""Analyze this data visualization and provide 
                    an EXECUTIVE SUMMARY format.

                    Chart Type: {chart_type.upper()}
                    Why: {recommendation_reason}
                    
                    {data_summary}
                    
                    REQUIRED FORMAT:
                    
                    ## Executive Summary
                    [2-3 narrative paragraphs synthesizing critical insights]
                    
                    ## Key Findings
                    [3-5 substantive insights as complete sentences]
                    
                    ## Detailed Analysis
                    ### Trend Analysis
                    [2-3 paragraphs on patterns over time]
                    
                    ### Comparative Insights
                    [2-3 paragraphs comparing categories]
                    
                    ### Notable Observations
                    [1-2 paragraphs on anomalies]
                    
                    ## Recommendations
                    [2-4 paragraphs with actionable advice]
                    
                    Use narrative paragraphs, not bullets. Be specific 
                    with numbers. Write for senior leadership."""
                }
            ],
        }]
    )
    
    return message.content[0].text

What we get back is remarkable. Claude doesn’t just describe what it sees—it interprets it. For that NFL data, Claude might notice that the 2008 financial crisis caused a synchronized drop across all divisions (even though we never told it what happened in 2008!). It might observe that the AFC East has been unusually consistent compared to other divisions. It might recommend strategic approaches based on volatility patterns.

This is the narrative we were looking for. The story behind the numbers.

The Real-World Test: From Concept to Practice

Let me give you a concrete example of how this works in practice. Say you’re analyzing Stanford’s research funding over the past 20 years across different departments. You upload a CSV:

Department,2004,2005,2006,...,2023
Computer Science,5.2M,5.8M,6.1M,...,15.3M
Biology,8.1M,8.4M,8.9M,...,12.2M
Physics,6.5M,6.3M,6.8M,...,8.1M

Here’s what happens, step by step:

Step 1: Claude analyzes the structure and says, “LINE: Multiple departments tracked over two decades—perfect for showing trends over time.”

Step 2: The system generates a line chart with three distinct, colorblind-friendly lines showing each department’s trajectory.

Step 3: Claude analyzes the chart and might write something like:

“The research funding landscape at Stanford reveals a remarkable transformation over two decades. Computer Science funding has nearly tripled from $5.2M to $15.3M, reflecting the university’s strategic pivot toward technology and the broader tech boom in Silicon Valley. Biology, while still well-funded, has seen comparatively modest growth, suggesting either maturity in the field or shifting institutional priorities. Physics funding has remained relatively flat, which in inflation-adjusted terms represents a decline in real resources…”

See how it’s not just describing the lines? It’s connecting dots. It’s providing context. It’s telling you what it means.

The Stanford Touch: Branding as a Feature

Now, you might think branding is just about making things look pretty. And sure, the Stanford Cardinal Red (#8C1515) throughout the interface does look sharp. But there’s a deeper reason I spent time on this.

When you’re presenting data analysis to university leadership, the Board of Trustees, or potential donors, the tool itself becomes part of your credibility. If your analysis looks like it came from a generic template, people unconsciously discount it. But when it’s professionally branded, when it looks like it belongs at Stanford, suddenly your insights carry more weight.

This is something I learned from my Apple days—design isn’t just aesthetics, it’s communication. The Cardinal Red bar at the top of the page, the professional typography, the clean layout—these things say “this is serious work from a serious institution.”

Here’s a snippet of the CSS that makes this happen:

.stanford-bar {
    background: #8C1515;  /* Cardinal Red */
    height: 8px;
    width: 100%;
}

.header {
    background: white;
    padding: 40px;
    border-top: 4px solid #8C1515;
}

.stanford-logo h1 {
    color: #8C1515;
    font-size: 2.8em;
    font-weight: 600;
    letter-spacing: -0.5px;
}

.btn {
    background: #8C1515;
    color: white;
    text-transform: uppercase;
    letter-spacing: 0.5px;
}

Simple CSS, but it transforms the entire feel of the application. It’s the difference between “some student project” and “Stanford’s analytics platform.”

The Elephant in the Server Room: AI Hallucinations

Now let’s talk about the thing that keeps AI researchers up at night: hallucinations. And I’m not talking about the good kind you might experience at a Grateful Dead concert. I’m talking about when AI confidently tells you things that are completely wrong.

Here’s the uncomfortable truth: Claude, GPT-4, all these large language models—they can make stuff up. They might tell you there was a major regulatory change in 2015 that affected your sales, when no such thing happened. They might confidently cite statistics that don’t exist. And the scary part? They’ll do it in the same confident, professional tone they use when they’re absolutely right.

This isn’t a bug—it’s a fundamental characteristic of how these models work. They’re predicting what words should come next based on patterns in their training data, not reasoning from first principles or checking facts against a database.

So how do we deal with this in Stanford DataSage?

Strategy #1: Ground the AI in Reality

We never ask Claude to recall facts from its training. Instead, we show it the actual data and the actual chart. Look at our prompt again:

message = client.messages.create(
    model="claude-sonnet-4-5-20250929",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {"data": chart_base64}},  # The actual chart
            {"type": "text", "text": f"{data_summary}"}           # The actual numbers
        ]
    }]
)

We’re not asking “What happened to Ford F-150 sales in 2008?” We’re saying “Here’s a chart showing Ford F-150 sales from 1990-2023, and here are the exact numbers. What patterns do you see?”

This is crucial. Claude is analyzing what we’re showing it, not trying to remember facts. It’s like the difference between asking someone “What did the weather report say yesterday?” versus showing them yesterday’s weather report and asking “What does this tell us about the forecast?”

Strategy #2: Make It Observable

Every analysis includes the data summary right there in the prompt. When Claude says “The F-150 averaged 823,000 units from 2000-2008,” you can verify that because we’ve included those numbers. Users can check the work.

In the frontend, we show both the chart AND the analysis. If Claude says something that doesn’t match what you see in the chart, it’s immediately obvious. This is like when your math teacher made you show your work—it makes errors detectable.

Strategy #3: Structure Reduces Fiction

Notice how we give Claude a very specific format to follow? That’s not just for aesthetics. When you give an AI a clear structure, it’s less likely to go off on creative tangents.

Compare these two prompts:

Bad:  "Tell me about this sales data."
Good: "Provide: 1) Executive Summary, 2) Key Findings, 3) Trend Analysis, 
       4) Comparative Insights, 5) Recommendations"

The second one keeps Claude focused. It’s like giving someone an outline to fill in rather than asking them to write an essay from scratch. Less room for imagination, more focus on analysis.

Strategy #4: The Human in the Loop

This is the most important strategy: Stanford DataSage is a tool, not an oracle. We’re not replacing human judgment—we’re augmenting it.

Think of it like spell-check. Spell-check is usually right when it suggests corrections, but sometimes it’s hilariously wrong. You still need to read what you write. Same principle here. Claude’s analysis is usually insightful, but you need to read it critically, especially if you’re going to present it to your Board of Directors.

The analysis is a starting point for discussion, not the final word. It’s meant to spark ideas, highlight patterns you might have missed, and save you hours of manual analysis. But the human being still needs to apply domain knowledge, consider context, and make the final judgment calls.

The Honest Assessment

Could Claude hallucinate and say something wrong about your data? Yes, theoretically. In practice, we’ve found it to be remarkably accurate when analyzing visual data it can actually see. But should you blindly copy-paste Claude’s analysis into your annual report without reading it? Absolutely not.

This is a proof of concept, remember. We’re exploring what’s possible when you combine BI with AI. Part of that exploration is understanding the limitations and building in safeguards. In a production system, you might add:

  • Confidence scores on statements
  • Automatic fact-checking against the raw data
  • Multiple model consensus (ask GPT-4 and Claude, compare answers)
  • User feedback loops to improve accuracy over time

But for now, the best safeguard is the same one we’ve always had: human intelligence reviewing machine intelligence.

The Engineering That Makes It Invisible

There’s a lot of engineering underneath this that users never see, but it’s worth talking about because it’s where a lot of the “just works” magic happens.

Dynamic Data Handling

The biggest challenge was making this work with any CSV file. Not just car sales data or NFL data, but anything. That meant building smart column detection:

# Figure out where labels end and data begins
id_columns = []
for col in df.columns:
    try:
        # Try to convert to numbers
        pd.to_numeric(df[col], errors='raise')
        break  # Found the data!
    except (ValueError, TypeError):
        id_columns.append(col)  # This is a label

This little loop is doing a lot of work. It’s walking through columns left-to-right, trying to convert each one to numbers. The first column that successfully converts? That’s where the data starts. Everything before it? Labels.

This means it works whether you have:

  • 1 label column (Product, 2020, 2021, 2022…)
  • 2 label columns (Make, Model, 1990, 1991…)
  • 3 label columns (League, Conference, Division, 1980, 1981…)
  • Or even more!

Type Juggling

CSV files are tricky. Sometimes numbers come in as strings. Sometimes you have quoted values. Sometimes column headers are integers, sometimes they’re strings. We need to handle all of it:

# Convert everything to the right types
year_labels = [str(col) for col in year_columns]  # Labels are strings
values = pd.to_numeric(row[year_columns], errors='coerce').values  # Data is numbers

# 'coerce' means "try your best, if it fails, use NaN"
# This handles bad data gracefully

That errors='coerce' parameter is doing a lot of heavy lifting. It means “convert to numbers if you can, but if you can’t, just put NaN (not a number) and keep going.” This graceful failure handling is what makes the system robust.

Base64 Image Encoding

Here’s a neat trick: instead of saving the chart as a file on disk and then serving it up, we encode it directly as base64 and send it in the JSON response:

buffer = BytesIO()  # Memory buffer, not disk
plt.savefig(buffer, format='png', dpi=150)
buffer.seek(0)
image_base64 = base64.b64encode(buffer.read()).decode()

return jsonify({
    'chart': image_base64,  # Send it right in the response
    'analysis': analysis
})

This means:

  • No temporary files cluttering up the server
  • No file cleanup needed
  • Faster response times
  • Simpler architecture

On the frontend, we just slap it into an img tag:

chartImage.src = 'data:image/png;base64,' + data.chart;

Boom. Chart appears. Magic.

Markdown to HTML

Claude returns its analysis in Markdown format (those ## headers, bold text, etc.). But browsers don’t natively render Markdown. So we use the marked.js library to convert it on-the-fly:

// Raw markdown from Claude
const markdownText = "## Executive Summary\n\nThe data shows..."

// Convert to HTML
analysisText.innerHTML = marked.parse(markdownText);

And then our CSS makes it look Stanford-professional:

.analysis-content h2 {
    color: #8C1515;
    font-size: 1.5em;
    border-bottom: 2px solid #8C1515;
    padding-bottom: 8px;
}

.analysis-content strong {
    color: #8C1515;
    font-weight: 700;
}

All the formatting happens automatically. Users just see a beautiful, professional report.

Performance: The Numbers Behind the Magic

Let’s talk speed and cost, because in the real world, these things matter.

Timing Breakdown:

  • Upload & Parse CSV: ~0.5 seconds
  • Claude chart recommendation: ~2-3 seconds
  • Generate visualization: ~1-2 seconds
  • Claude analysis: ~5-8 seconds
  • Total: ~10-15 seconds

That’s fast enough to feel real-time but not so instant that it feels like it’s not doing real work. It’s the Goldilocks zone of response time.

Cost per Analysis:

  • Chart recommendation: ~500 tokens = $0.0008
  • Analysis: ~3,500 tokens = $0.006
  • Total: ~$0.007 per upload

Less than a penny per complete analysis. Even if you’re analyzing hundreds of datasets, you’re talking a few dollars. That’s democratization—when the cost barrier essentially disappears.

Scaling Considerations:

Right now, this runs on a single Flask instance. For a proof of concept, that’s fine. But what if Stanford wanted to deploy this university-wide?

  • Add Redis for caching (same data gets same chart, no re-computation)
  • Use Celery for background jobs (don’t make users wait for API calls)
  • Load balance across multiple servers
  • Add rate limiting to prevent API quota issues

But here’s the thing—even without any of that, this could handle dozens of concurrent users easily. Flask might be simple, but it’s not slow.

Real-World Applications: Beyond the Proof of Concept

So we’ve built this thing. It works. It’s cool. But what’s it actually for?

Academic Research

Imagine a PhD student studying climate data. They’ve got decades of temperature readings, precipitation levels, CO2 measurements. They’re looking for patterns, trying to tell a story about climate change. Stanford DataSage could help them:

  1. Quickly visualize multiple variables
  2. Get AI-generated hypotheses about patterns
  3. Produce publication-ready charts
  4. Generate narrative descriptions for their papers

Instead of spending days in Excel, they get a starting point in seconds.

Administrative Decision-Making

Stanford’s administration deals with data all the time. Enrollment trends, budget allocations, facility usage, research output. Stanford DataSage could:

  1. Turn raw spreadsheets into executive presentations
  2. Identify concerning trends automatically
  3. Generate reports for Board meetings
  4. Make data accessible to non-technical decision-makers

The Provost shouldn’t need a data science degree to understand enrollment patterns.

Business Intelligence

For startups coming out of Stanford (and there are a lot of them), this approach could transform how they handle analytics:

  1. Revenue tracking across product lines
  2. Customer acquisition cost trends
  3. Churn analysis
  4. Market share evolution

The founder who’s brilliant at product development but terrible at spreadsheets now has a tool that speaks their language.

Educational Tool

This could be used in statistics and data science courses to help students understand:

  1. How to choose appropriate visualizations
  2. What makes a good data analysis
  3. How AI can augment human intelligence
  4. The importance of data storytelling

Students learn by example—seeing Claude’s analysis helps them understand what good analysis looks like.

The Future: Where This Could Go

This is a proof of concept, which means it’s really just scratching the surface. Here’s what keeps me up at night thinking about possibilities:

Interactive Exploration

Right now, you upload data, you get analysis. Done. But what if you could ask follow-up questions?

“Claude, what happened in 2008 that caused that drop?” “Show me just the top three performers.” “How does this compare to industry benchmarks?”

The technology exists. You’d need to maintain conversation state and allow iterative refinement. Totally doable.

Multiple Data Sources

What if instead of uploading one CSV, you could connect to:

  • Your university’s data warehouse
  • Google Sheets
  • SQL databases
  • Real-time APIs

Now you’re not just analyzing static data—you’re building a living dashboard that updates automatically.

Collaborative Analysis

Imagine multiple people working on the same dataset:

  • One person uploads the data
  • Another person adds context and annotations
  • A third person challenges the AI’s conclusions
  • Together they refine the narrative

This becomes a collaborative intelligence tool, not just an individual one.

Custom Training

Right now, we’re using Claude’s general knowledge. But what if we fine-tuned a model on Stanford-specific data and terminology? It could understand university jargon, know about historical context, recognize patterns specific to higher education.

The model would be smarter because it’s specialized.

Prediction and Forecasting

We’re analyzing historical data. But Claude (or other AI models) could also forecast:

  • “Based on these trends, here’s what I predict for next year”
  • “If this pattern continues, you’ll reach your target by Q3 2025”
  • “This trajectory is unsustainable; consider changing course”

From understanding the past to predicting the future.

The Philosophical Bit: Why This Matters

I’ve been in tech since Buleiten Board Systems were every teenage boy’s dream. More on the anicarst cookbook in another post. I’ve seen a lot of technologies come and go. And I’ve learned that the ones that matter aren’t necessarily the most technically impressive—they’re the ones that empower people.

Being an Apple enthusiast, I think back to the Apple II; it wasn’t the most powerful computer. But it brought computing to everyday people. The Macintosh wasn’t the first computer with a GUI. But it made computers friendly and approachable.

That’s what I think is exciting about combining BI with AI. We’re not replacing data analysts. We’re not automating jobs. We’re making data analysis accessible to people who aren’t specialists. We’re democratizing insight.

Your average department head at Stanford isn’t going to learn Python and Pandas. They shouldn’t have to. But they do need to understand their department’s performance data. Stanford DataSage bridges that gap.

Data Literacy for Everyone

There’s this concept in education called “scaffolding”—you provide support structure that helps people learn something new, then gradually remove the support as they gain mastery.

That’s what this tool does. The AI provides the scaffolding:

  • “This is what a line chart is good for”
  • “Here’s what this pattern means”
  • “This is how you should think about this data”

Over time, users start to internalize these lessons. They become more data-literate. The scaffolding becomes less necessary. But it was crucial for getting them started.

Trust but Verify

Remember when calculators came out and teachers were worried students wouldn’t learn math? The concern was valid—but misplaced. Students still needed to understand math to know if the calculator’s answer made sense. The calculator just freed them from tedious arithmetic so they could focus on problem-solving.

AI for data analysis is similar. You still need to think critically about what the AI is telling you. But you’re freed from the tedium of manually calculating trends and looking for patterns. You can focus on the meaning, the implications, the decisions.

The Human-AI Partnership

This is the future of work: humans and AI working together, each doing what they’re good at.

AI is good at:

  • Processing large volumes of data quickly
  • Spotting patterns humans might miss
  • Generating structured text
  • Staying patient with repetitive tasks

Humans are good at:

  • Understanding context and nuance
  • Applying domain expertise
  • Making judgment calls
  • Asking “why does this matter?”

Stanford DataSage puts these together. The AI does the grunt work. The human does the thinking.

Technical Deep Dive: The Complete Stack

For the engineers and developers reading this (hey, fellow nerds!), let’s talk about the complete technical stack:

Backend:

Flask==3.0.0          # Web framework - simple, flexible, perfect for this
pandas==2.1.3         # Data manipulation - the NumPy of dataframes
matplotlib==3.8.2     # Visualization - old reliable
anthropic==0.39.0     # Claude API - the AI brain
numpy==1.24.3         # Numerical operations - pandas depends on it

Frontend:

// Pure vanilla JavaScript - no frameworks needed
// marked.js for Markdown rendering
// That's it. Keep it simple.

Architecture Pattern:

User → Upload CSV → Flask Backend → {
    1. Pandas reads/processes data
    2. Claude recommends chart type
    3. Matplotlib generates chart
    4. Chart encoded as base64
    5. Claude analyzes chart + data
    6. Return JSON with chart + analysis
} → Frontend displays results

Key Design Decisions:

  1. Why Flask instead of Django?

    • Simpler for our use case
    • Less boilerplate
    • Easier to understand for newcomers
    • Perfect for a proof of concept

  2. Why not use a JavaScript charting library?

    • Matplotlib gives us more control
    • Server-side rendering means consistent output
    • Base64 encoding simplifies architecture
    • No client-side dependencies

  3. Why vanilla JavaScript instead of React?

    • Only one page, no complex state management needed
    • Faster load time
    • Easier for non-frontend developers to understand
    • Fewer dependencies = fewer things to break

  4. Why Claude instead of GPT-4?

    • Excellent at structured output
    • Strong vision capabilities
    • Good at following complex instructions
    • Anthropic’s focus on safety and accuracy

The Complete Flask App Structure:

from flask import Flask, render_template, request, jsonify
import pandas as pd
import matplotlib
matplotlib.use('Agg')  # No GUI backend needed
import matplotlib.pyplot as plt
import anthropic
import base64
from io import BytesIO
import os

app = Flask(__name__)
app.config['UPLOAD_FOLDER'] = 'uploads'
app.config['MAX_CONTENT_LENGTH'] = 16 * 1024 * 1024  # 16MB max

ANTHROPIC_API_KEY = os.environ.get('ANTHROPIC_API_KEY')

@app.route('/')
def index():
    """Serve the main page"""
    return render_template('index.html')

@app.route('/upload', methods=['POST'])
def upload_file():
    """Handle file upload and orchestrate analysis"""
    try:
        # Validate file
        if 'file' not in request.files:
            return jsonify({'error': 'No file provided'}), 400
        
        file = request.files['file']
        if not file.filename.endswith('.csv'):
            return jsonify({'error': 'Only CSV files allowed'}), 400
        
        # Process data
        df = pd.read_csv(file)
        
        # Get AI recommendation
        chart_type, reason = get_chart_recommendation(df)
        
        # Generate visualization
        chart_base64 = generate_chart(df, chart_type)
        
        # Get AI analysis
        analysis = get_claude_analysis(chart_base64, df, chart_type, reason)
        
        # Return everything
        return jsonify({
            'success': True,
            'chart': chart_base64,
            'analysis': analysis,
            'chart_type': chart_type,
            'recommendation': reason
        })
    
    except Exception as e:
        return jsonify({'error': str(e)}), 500

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port=5000)

Clean. Simple. Does one thing well.

Lessons Learned: What Worked and What Didn’t

Building this taught me some things:

What Worked:

  1. Simple > Complex: Started with basic features, added complexity gradually. If I’d tried to build everything at once, I’d still be building.

  2. User Testing Early: Showed it to people who weren’t engineers. Their “I don’t understand” feedback was invaluable.

  3. Embrace Constraints: Limited to Claude’s API? That actually made design decisions easier.

  4. Document as You Go: Writing docs while building kept the “why” fresh in my mind.

What Didn’t:

  1. First Colorblind Approach: Initially used Stanford colors for chart data. Looked pretty, failed accessibility tests. Had to redo it.

  2. Overly Complex Prompts: First version of the analysis prompt was too prescriptive. Claude gave wooden, formulaic responses. Had to simplify and trust the model more.

  3. Assuming CSV Standards: Thought I could rely on CSVs being formatted consistently. Nope. Had to add a ton of error handling.

  4. Not Testing Edge Cases: First version broke on single-column data, empty files, all-text columns. Had to add defensive coding.

The Biggest Surprise:

Claude was better at choosing chart types than I expected. I thought I’d need to give it lots of rules and guidance. But its general understanding of data visualization principles was already solid. Sometimes less instruction leads to better results.

The Ethics: Responsible AI Integration

We need to talk about responsibility, because building AI tools isn’t just about what we can do—it’s about what we should do.

Transparency:

Users need to know they’re getting AI-generated analysis. That’s why the interface clearly labels it as “Claude AI Analysis” with the Claude badge. No hiding the fact that a machine wrote this.

Accountability:

The analysis includes the data summary, so users can verify claims. If Claude says “averaged 823,000 units,” you can check that. This is accountability through transparency.

Bias Awareness:

AI models can perpetuate biases present in their training data. In our case, we’re analyzing user-provided data, so the bias risk is lower. But Claude’s interpretation could still reflect biases. Users need to be aware and critical.

Access and Equity:

By making this free (or at least very cheap) and web-based, we’re reducing barriers to entry. You don’t need expensive software licenses. You don’t need technical expertise. That’s important for equity.

Environmental Consideration:

AI models consume electricity. Each API call has a carbon footprint. We’ve optimized to minimize redundant calls, but it’s worth being conscious of. Sometimes the old way (manual analysis) is the greener way.

The Job Displacement Question:

I’m often asked: “Will this replace data analysts?” And my answer is the same as it was when spreadsheets came out: No, but it will change what they do.

Data analysts will spend less time on routine visualization and more time on:

  • Deep domain-specific analysis
  • Strategic thinking
  • Data quality and collection
  • Custom modeling
  • Communicating with stakeholders

The tool augments; it doesn’t replace.

How to Actually Use This (A Practical Guide)

Alright, enough philosophy. Let’s talk about how you’d actually use this thing.

Step 1: Get Your Data Ready

Your CSV should look something like this:

Category,Subcategory,2020,2021,2022,2023
Product A,Type 1,1000,1100,1200,1300
Product B,Type 2,1500,1400,1600,1700

First columns: Labels (text) Later columns: Data (numbers)

Step 2: Upload

Just drag and drop. Or click to browse. The interface will tell you if something’s wrong with your file.

Step 3: Wait 10-15 Seconds

Go grab a coffee. Or don’t—it’s pretty quick.

Step 4: Review the Recommendation

Look at what chart type Claude chose and why. Does it make sense? If you disagree, that’s interesting—think about why.

Step 5: Examine the Chart

Look at the visualization. Does anything jump out at you? Any patterns, trends, or anomalies?

Step 6: Read the Analysis

Start with the Executive Summary. Get the big picture. Then dive into details if you need them.

Step 7: Verify

Check Claude’s numbers against your data. Make sure the interpretation makes sense given what you know about the domain.

Step 8: Act

Use the insights. Make decisions. Create presentations. Write reports. This is the tool—you’re the craftsperson.

The Meta-Lesson: Building AI Tools Responsibly

Here’s what I think is the most important takeaway from this whole project: AI should augment human capability, not replace human judgment.

Every design decision in Stanford DataSage reflects this principle:

  • We show the data alongside the analysis (so you can verify)
  • We explain which chart type was chosen (so you understand the reasoning)
  • We structure the output (so you can navigate to what matters to you)
  • We keep the human in the loop (you still have to read and evaluate)

This is how I think AI should be integrated into tools. Not as a black box that spits out answers you’re supposed to trust. But as a collaborator that does the heavy lifting while you do the thinking.

The Beginning, Not the End

So that’s Stanford DataSage. A proof of concept for merging business intelligence with artificial intelligence. A demonstration that we can make data analysis more accessible without dumbing it down. A tool that tells stories from numbers.

Is it perfect? No. Will it hallucinate occasionally? Probably. Should you blindly trust it? Absolutely not.

But that’s okay. Because the goal isn’t perfection—it’s progress. We’re learning what works, what doesn’t, and how to build AI tools that actually help people.

I built this in a few days with freely available tools. The AI API costs pennies per use. The code is straightforward enough that a CS undergraduate could understand it. That’s the real magic here—not any individual piece of technology, but the fact that these powerful capabilities are now accessible to anyone with curiosity and determination.

We’re taking something that seems complex and intimidating, and we’re making it approachable. We’re putting power in people’s hands.

That’s what gets me excited about technology. Not the algorithms or the architectures—though those are fun too. But the democratization. The empowerment. The moment when someone who thought they couldn’t do something suddenly realizes they can.

So if you’re working on integrating AI into your domain, here’s my advice:

  1. Start simple: Proof of concept beats perfect product.
  2. Stay grounded: AI should analyze real data, not hallucinate.
  3. Be transparent: Show your work, explain your reasoning.
  4. Keep humans central: Augment judgment, don’t replace it.
  5. Think about access: Make it available to everyone.

And most importantly: Build things that actually help people.

Because at the end of the day, that’s what matters. Not how clever your algorithm is, or how many parameters your model has, or how low your API costs are. But whether you’ve made someone’s life better, easier, or more productive.

That’s the metric that counts.

Now go build something cool.

P.S. - If you want to try “Stanford DataSage” the code is available on code.stanford.edu. And if you build something interesting with it, drop me a line. I’d love to hear what you create.

P.P.S. - Remember: AI is a tool. You’re the craftsperson. Use it wisely.

Technical Stack: Python 3.8+, Flask, Pandas, Matplotlib, Claude API (Anthropic), HTML/CSS/JavaScript

Acknowledgments: Thanks to the Anthropic team for Claude, the matplotlib community for making data visualization accessible, and Stanford University for being a place where proof-of-concepts are encouraged.

“The best way to predict the future is to invent it.” — Alan Kay (but Woz probably said something similar at some point)