I swear none of you guys have even attempted to use AI to do data analysis. I have, I built a MCP and integrated a copilot agent into Teams which has access to specific database data, and refined the rules for it to the point where the CFO rigorously tested it (and still does) and trusts the results it returns.
The problem with being a pragmatic LLM user is that you have on one side corporate America shoe-horning the tech in mediocre products none wants, and on the other side a large portion of the internet who loathe it but don’t use it and don’t even know what it does. Those conversations never go anywhere man. You’re talking to someone who thinks accuracy of 57% on SpreadsheetBench means the model gives wrong answers 42% of the time.
Hate to agree with Microsoft but yeah, Excel is probably a great place to introduce an LLM. It’s in that sweet spot between natural language and light programming, in an environment with math baked in so you don’t really care about the model’s accuracy or exact recall. All the data is here, and the model only has to manipulates cell numbers and writes formulas in this dumbed down language.
I’m sure you can get away with pretty small models too. It doesn’t need super human knowledge to implement 90% of common Excel use cases, and i suspect in real world scenarios the accuracy must be pretty interesting.
Until it starts pulling data from nonexistent worksheets
You tell it not to.
I swear none of you guys have even attempted to use AI to do data analysis. I have, I built a MCP and integrated a copilot agent into Teams which has access to specific database data, and refined the rules for it to the point where the CFO rigorously tested it (and still does) and trusts the results it returns.
The problem with being a pragmatic LLM user is that you have on one side corporate America shoe-horning the tech in mediocre products none wants, and on the other side a large portion of the internet who loathe it but don’t use it and don’t even know what it does. Those conversations never go anywhere man. You’re talking to someone who thinks accuracy of 57% on SpreadsheetBench means the model gives wrong answers 42% of the time.
Hate to agree with Microsoft but yeah, Excel is probably a great place to introduce an LLM. It’s in that sweet spot between natural language and light programming, in an environment with math baked in so you don’t really care about the model’s accuracy or exact recall. All the data is here, and the model only has to manipulates cell numbers and writes formulas in this dumbed down language.
I’m sure you can get away with pretty small models too. It doesn’t need super human knowledge to implement 90% of common Excel use cases, and i suspect in real world scenarios the accuracy must be pretty interesting.