Approach

Decide it's easy, then make it easy

The first half is a trick I play on myself. 'This is hard' is the thought that stops me starting, and while it sits there my effort goes on the size of the problem instead of on the problem. Deciding it is easy gets me moving. The second half is the part I have to earn: the only way to keep that promise is to break the thing down until the pieces genuinely are easy. I have done that enough times now that most things do look easy to me. That is the trick working, not the work being small.

It also means I think around a problem rather than straight at it, because a route I have decided is easy is one I will actually try. And I have almost always been the only data person in the room, which leaves no team to absorb a bad abstraction and nothing I ship maintained by anyone but me. Five decisions that come out of working this way:

Start at the constraint, not the schema

The default: Model the domain first (entities, tables, relationships) and work out later what anyone actually needs from it.
The problem: I can spend a month building the wrong thing very precisely. Nobody on a production floor ever asked me for a schema; they asked for a plan they could trust, and a tidy model is not evidence that I understood the work.
What I do: I find the one rule the work genuinely turns on before I design anything, and I find it next to the people doing the work rather than in their data. On the planning engine that rule was a sentence a planner said out loud, 'a full week is full', and every table, parameter and Spark job after it exists to enforce that sentence. They did not stop being involved once I had it: the eighty parameters and twelve rule tables they edit are the same levers we argued over at the whiteboard, so the engine keeps changing in their hands rather than mine. The cost is that I start slower and my first data model is usually worse than it should be.

Whatever will keep changing shouldn't be code

The default: Business rules get written into the code that applies them, because that is the easiest and fastest place to put them.
The problem: Every later change to a rule then needs me, a deployment and a release window. And these rules change constantly, not because they were wrong, but because the business moved. I become a queue that the business has to wait in.
What I do: I try to work out early which parts will keep moving, and put those in data that someone else is allowed to edit. In the scheduler that is roughly eighty parameters and twelve rule tables the planners change themselves, with zero deployments. It is genuinely harder to build and much harder to test, which is why it has an automated harness that rejects an invalid rule change rather than quietly producing a bad plan.

A system should be allowed to refuse

The default: An automation is judged on how much it gets through without a human: the coverage number is the headline.
The problem: Chasing coverage pushes a system into guessing, and a confident wrong answer is worse than no automation at all. The manual process had somebody checking; the automated one does not. Being wrong two percent of the time destroys trust in the other ninety-eight.
What I do: I give it a confidence threshold and a way to say I don't know. The invoice pipeline holds a document for a human rather than writing a sales order against an item number it is unsure of. It automates less than it could. It gets believed, which is the thing that actually determines whether anyone keeps using it.

Data layers are about who is allowed to be wrong

The default: Bronze, silver and gold are taught as three stages of progressively cleaner data: raw, tidied, ready.
The problem: Described as a cleaning process it sounds like housekeeping, and it gave me no way to decide what belonged in which layer. On the first platform I built, I could not tell.
What I do: I think about permission to be wrong. Bronze is allowed to be wrong: it holds whatever the source really sent, faithfully, including the mistakes. Gold is not allowed to be wrong, because people make decisions on it. Silver is where that gets corrected, deliberately and visibly. Once the layers mean that, a disagreement about a number stops being an argument and becomes a lookup: we can point at the layer where it went wrong.

I ship the interface too

The default: A data engineer delivers correct tables, and somebody else builds the thing people actually look at.
The problem: The handover is where most data work quietly dies. A number nobody can see is a number nobody trusts, and if I never watch someone use it I never find out that the answer I was so pleased with is unusable at 7am on a Monday.
What I do: I take it all the way to the screen: Spark job, API, and the page the planner opens. What this costs is real and worth saying plainly: at each individual layer I am worse than a specialist would be. What I get for it is that nothing is lost in a handover, and it is the only way one person covers this much ground.

See the work Writing Get in touch