The pattern is familiar by now. A team stands up a data lake, points every source system at it during data lake architecture planning, and now nobody trusts what’s in it. Metadata is missing, data ownership was never assigned to anyone, data quality is inconsistent domain to domain, and what started as a promising asset has quietly become a data swamp. The lake didn’t fail because the storage was wrong. It failed because data lake requirements were never defined before building started, and requirements gathering got skipped in the rush to start ingesting.
Requirements for a data lake mean something broader than storage capacity or which cloud to use. They mean the business outcomes the lake needs to support, the data domains it will cover, the governance and security controls wrapped around it, and the operating model that keeps it healthy after launch. Skipping straight to architecture without answering those questions first is how a lakehouse and a data warehouse end up conflated, when in practice a lake is meant to hold raw and curated data at scale while a warehouse serves governed BI, and a lakehouse pattern only makes sense once you know which of those problems you’re solving.
Start from Decisions, not Data
The strongest data platform work we’ve done starts by working backward from the decisions a business actually needs to make, not from an inventory of available data. Pricing, inventory, churn, fraud, whatever the priority use cases are, each one should be ranked by ROI and feasibility before a single pipeline gets built, and each should translate into a concrete data product with a defined SLA and SLO consumers can rely on. This is where we push back hardest on the instinct to ingest everything by default. Every dataset needs a reason to be there and an owner accountable for it, or it becomes exactly the kind of unowned sprawl that turns a data lake into a data swamp.
That discipline, an assessment-first approach that identifies gaps and opportunities before recommending any solution, is what separated our work on ATCS’s Databricks lakehouse from a typical migration. Rather than pointing every source at raw storage in a landing zone and hoping structure would emerge later, we mapped the priority decisions the platform needed to support first, then built the medallion architecture, raw data landing in that zone, cleaned and modeled data moving into curated zones through defined orchestration, around those decisions specifically. Unity Catalog, our data catalog of choice, sat across the whole structure from day one, not bolted on after the fact, which is exactly why the infrastructure is in place today rather than half-finished with definitions nobody agreed on.
Once the priority use cases are clear, the next requirement is a real inventory of source systems, CRM, ERP, operational databases, SaaS applications, files, logs, each with its own API limits and its own options for streaming ingestion or CDC, short for change data capture, versus simple batch ingestion. Grouping those sources into domains, customer, product, finance, operations, and assigning a steward to each one, is what prevents the same customer record from existing in three inconsistent forms across the lake, and it’s the foundation of real data stewardship rather than governance in name only.
Only after that mapping is done does it make sense to decide on zones and file formats. A raw zone, a staged zone, a curated zone, and often a sandbox for exploration, each with defined naming conventions and a partitioning strategy and dataset versioning so nothing gets overwritten silently. File and table format choices, Parquet, ORC, or Avro underneath, and Delta Lake, Apache Iceberg, or Apache Hudi on top for ACID transactions and schema evolution, follow from the domains and use cases, not the other way around.
Whether data should be schema-on-read for exploration or schema-on-write for trusted, governed datasets is part of that same decision.
Governance & Security
Trust in a data lake is a feature, not an afterthought, and that means data quality checks, data lineage tracing every field from source through transformation to the dashboards and machine learning models consuming it, and audit logs covering both access and changes, need to exist from day one, backed by real observability and monitoring and alerting rather than discovered after an incident.
Security and compliance requirements are non-negotiable in the same way. An identity and access model, IAM, built on least privilege, with encryption at rest and encryption in transit and real key management through KMS behind it. Row-level security, column-level security, data masking, and tokenization to protect PII, mapped against whatever applies to your data, GDPR, CCPA, HIPAA, PCI DSS, SOC 2, along with clear data retention schedules, defined data residency, and a working right-to-delete process for subject requests.
Start Small
The implementation plan that works starts small on purpose. One domain, one end-to-end use case, with governance, cataloging, and security included from day one rather than promised for later. That minimum scope, delivered with clear SLOs that differ appropriately between raw and curated datasets, and disaster recovery targets defined through RPO and RTO, is what proves the operating model before scaling it out domain by domain. FinOps guardrails, chargeback or show back by team, and open source tools where they fit belong in that first phase too, since cost control retrofitted after the lake is already large is a much harder problem than cost control designed in from the start.
None of this is about picking the trendiest table format or the newest orchestration tool. It’s about defining, before any of that gets chosen, what the lake needs to do, who owns each piece of it, and how you’ll know it’s working. That’s the same foundation-first question we bring to every data platform engagement, and it’s the difference between a lake that becomes durable infrastructure and one that quietly becomes something nobody wants to query.
