No video of the event yet, sorry!

Data in any organization is one of its most valuable assets. A modern solution is the 'Datalake' a centralized repository designed to store, process, and secure large amounts of structured, semistructured, and unstructured data.

As the datalake grows, complexity builds up, the data engineer faces many challenges:

• Diverse data sources spanning SQL databases, events, S3/Blob files, SFTP, and web crawling.
• Different environments or versions for the datalake.
• Data models and transformations.
• Data type validations and correctness.
• Visibility and monitoring.
• Orchestration.

Many teams end up having large and complex notebooks, scripts and pipelines, with little to no centralized definitions of the Data models, storages and environments, this eventually slows down development.

We faced this issue: more than a hundred models, terabytes of data, different storages/types, several environments for developing, complex QA and validations.

We decided that this problem needed a more Software Engineering oriented solution: developing our custom solution.

A bespoke framework to create models, data pipelines, handle the complexity of storages and instances and many of the aforementioned problems.

The solution became one of the cornerstones of the data team.

We re-wrote some pieces from scratch in an Open Source library written in Python and based on the popular and modern dataframe library Polars, join us as we expand on Data problems, Datalake challenges and some of the code we wrote to solve it.