Nobody gets excited about file formats... but you should.
The overlooked critical consideration in any software estate refresh.
Many of us would claim that we own our data, legally perhaps, but is it practically true? Let us consider a common valuation pipeline.
The policy administration system produces a delimited batch extract. That file is converted into the modelling platform's own format, because the platform cannot read the extract directly. The model runs and writes its results in a second proprietary format which your analytics tools now can't read. Another conversion, this time into something the data warehouse or the reporting tool can store.
Four stages, two format conversions, reconciliation requirements introduced at the transformations and complications in the data lineage. Each reconciliation is a control, which means each is also a piece of work and a potential point of failure. The format of the admin system extract often does not match the format of the results data storage which adds additional analysis overhead when querying across these files.
We've been designing around vendor closed formats for decades and the build up of drag is showing. The common format glue most commonly chosen is CSV, probably because CSV opens in anything. You can read it in Notepad, drop it into Excel and start pivoting. When three systems refuse to speak to each other, CSV is the diplomat that gets the conversation started. But CSV comes with some serious limitations. No schema enforced, no data type safety, no record of where it came from. The delimiter is a convention, the date format is a guess, the leading zeros in fixed length fields often get lost, commas creep into number formats which break the file structure. It is not so much a format as the absence of one.
For a long time vendor lock-in has been driven through data restriction rather than product effectiveness.
What are open formats and what problems do they solve?
What an open format is
An open file format is one whose structure is publicly documented and freely implementable. Anyone can write software that reads it or produces it, without a licence, a partner agreement or the original vendor's approval. HTML is open. PDF became open. PNG is open. The formats holding your model logic and your projection results, in most actuarial estates, are not.
Two formats to know by name
There are dozens of open formats, and a technical team can spend a long time comparing them. There is no need. Two have become the de facto standards, one for logic and one for data, and the sensible choice is to run on what the rest of the world already runs on. Both are recognised and supported by every mainstream tool. Both are well structured where structure counts. Both are dense and built for the way modern software reads and analyses information.
JSON (JavaScript Object Notation) is a text format for structured information. It is human-readable, universally parsed, and has become the default way software describes configuration and logic to itself. It suits the parts of your estate that describe how a calculation works: model definitions and formula logic, assumption basis configurations and workflow execution plans. A model expressed in JSON is a model you can open in a text editor, put under version control, compare line by line against last quarter's version, and validate against a schema.
Parquet is a binary, columnar format for data at scale. Where CSV stores one row after another, Parquet stores one column after another, with the data types and structure written into the file itself. The result is smaller files, faster queries directly with SQL, and a dataset that any modern analytical tool can read directly, from Python and DuckDB to Spark and Power BI, and yes, Excel. It suits the parts of your estate that are data: model points, assumptions and results outputs. Parquet deserves a post of its own and it will get one. For now it is enough to know that it exists, it is free, and the analytics world standardised on it some years ago while actuarial software mostly ignored it.
Six reasons to demand open formats
True Interoperability. Interoperability in most estates is achieved by people. Each handover is a job which introduces friction and slows everything down. When the model definition, model points, assumptions and results are freely accessible, efficient to access and commonly understood by tooling - everything throughout the process gets easier. Tasks are completed faster, fewer errors are made and your team is free to use the best tool for the job not the only one that works.
Shorter pipelines = improved lineage. Go back to the pipeline at the top of this post and run it on open formats. The model should read the admin extract directly. The model writes its results likewise into storage and analysis friendly formats. The warehouse, the analytics team and the reporting layer query those results where they sit. Two proprietary hurdles have disappeared and three implied reconciliations have collapsed to one: model output against model input.
Size is money. Parquet compresses natively, and columnar data compresses well, because a two million record column of product codes or valuation dates has very little variety in it. A CSV file typically shrinks to around a tenth of its size as Parquet, and because a query reads only the columns it needs, it reads less and returns faster. Less storage, less network, smaller infrastructure = lower cost. These savings are immediate in a cloud environment where you pay for what you use.
Analysis brings insights. Reducing the barriers to analyses means we can do more of them, more often. Writing to analytic-friendly formats directly from our modelling solution means we enable faster querying within and across runs. Parquet is commonly the format of choice for data lake storage and if the partitioning and cataloguing is used correctly then these analytic use cases will continue to scale well with volume. Inter-period and inter-basis analyses no longer rely on key person dependencies to configure.
AI ready. To optimise outcomes from agentic processes we should ensure that we make the process and data accessibility as simple as possible. Try asking your LLM to provide the file structure for a vendor specific format and observe how long it takes to answer, typically performing web searches and chewing through tokens (read money) in the process. Now ask it about an open file format such as Parquet and observe the difference. It replies instantly and broadly since open formats have good representation in the original agent datasets. Further to this, the agent has access to existing libraries such as pandas and polars for python which can be used instantly to analyse the data at scale. An agent cannot review a model it cannot read. It cannot trace a number through a binary. It cannot query results that only exist inside a vendor's interface. JSON and Parquet are the difference between an estate an agent can reason over and one it cannot.
Futureproof your process. Every tool in an actuarial estate will eventually be replaced. The pace of change today suggests that our software cycles are going to shorten dramatically into the future. The grid gives way to the cloud, the desktop tool to the platform, the platform to whatever AI eventually evolves into. What survives each transition is not the software. It is the core business logic and our history of data - we need to ensure that we decouple from vendor lock-in through data control which has been the Achilles heel of operational efficiency.
Three tests you can run this week
- Map your process for any key model and count the number of unnecessary transformations in the process.
- Can you run a third-party analysis tool directly on top of the results files produced by your models?
- How efficiently can your AI process interpret the data and maintain the lineage over a full valuation pipeline?
If you are considering AI readiness or a software refresh within your actuarial estate - open architecture and the use of open file formats should be at the top of your list of non-negotiables.