Files, tables, and the gap between them

Start with the thing everyone actually has: a folder in object storage with files in it. New ones arrive; nobody deletes the old ones.

You can query that folder. Point a query engine at it, and it will read every file and treat the union as one dataset. For a long time, that was the data lake — and it works, until it does not.

What breaks

A reader can see a half-written batch. A job writing today's files is not atomic. A query that runs mid-write reads some of them, produces a number, and that number is wrong in a way nothing reports.

You cannot change your mind. Updating one record means rewriting the file it lives in. Deleting one — for a legal request, say — means finding which of forty thousand files contains it.

Nothing agrees on the schema. File 900 has a column the first 899 do not. Some engines will fail, some will fill nulls, and which one you get depends on the reader.

You cannot see the past. Somebody asks why Tuesday's report differs from what they screenshotted on Tuesday. There is no answer available, because the folder only has a present tense.

What a table adds

A table is that same folder plus a transaction log: an ordered record of which files make up the table right now, and which made it up at every earlier point.

That one addition fixes all four:

• A writer adds files, then commits them to the log in one step. A reader that started earlier keeps reading the earlier version. Nobody sees a half-write.
• An update writes new files and commits a version where the old ones are no longer part of the table. The change is a log entry, not a rewrite of everything.
• The log records the schema, so a change is either rejected or evolved on purpose.
• Every past version is still described in the log, so "what did this look like on Tuesday" is a query rather than an apology.

That is what Delta is on Databricks, and it is worth understanding as this — the log — rather than as a file format. The files were never the hard part.

Why this is the first lesson

Nearly everything later in this course is a consequence of the log existing: incremental merges, time travel, the streaming reader that knows where it stopped, the expectations that reject a bad batch before it commits.

If the log makes sense now, most of what follows will feel inevitable rather than arbitrary.