You will work within a large organization. You will join a team that unlocks critical financial data from internal and external sources for operations and analysis purposes. The team is currently building a new analytics platform entirely in Azure Databricks and also manages several large cloud data warehouses. You will work closely with data engineers, analysts, and the platform team.
What you will do.
- Design, develop, and optimize data pipelines within Azure Databricks
- Implement data ingestion, transformation, and enrichment processes with Python, PySpark
- Set up and manage lakehouse architectures according to the medallion architecture
- Develop and maintain Databricks notebooks, workflows, and Unity Catalog structures
- Implement Infrastructure as Code and CI/CD processes within Azure DevOps
- Set up and execute unit, integration, and regression tests
- Apply software engineering best practices, including version control, code reviews, linting, and security scanning
- Collaborate with data engineers, analysts, and the platform team to deliver high-quality data solutions
Profile
You combine in-depth knowledge of Azure and Databricks with strong software engineering principles. You are proactive, structured, and accurate, and you have a genuine feel for data.
What they ask
- Minimum 5 years of work experience with Data Engineering in the Azure Cloud environment
- Minimum 5 years of work experience with Python and PySpark
- Minimum 5 years of work experience with SQL in an enterprise environment
- Minimum 3 years of work experience with Databricks notebooks, pipelines, and Unity Catalog
- Minimum 3 years of work experience with Infrastructure as Code and Azure DevOps CI/CD pipelines
- Minimum 2 years of work experience with data lakehouses (e.g., medallion structure)
- Demonstrable experience with testing (unit and regression tests)
- Demonstrable experience with Git, code linting, and code scanning
- Minimum HBO (higher professional education) thinking and working level
- Good written and oral skills in the Dutch language
Wishes.
- Experience with designing and building dimensional models
- Minimum 2 years of experience with Data Vault 2.0
- Minimum 3 years of experience with data engineering / BI within the financial sector
- Minimum 3 years of experience with Azure Data Factory
- Minimum 3 years of experience with Azure Synapse Analytics
- Databricks certification (Data Engineer Associate or Professional)
How to apply
View the full assignment text and application details once your tailored application is ready.
Order a tailored application to view the full assignment and application details.
More context, less searching.
You get enough context to judge whether this job is relevant. The full brief, client details and next steps stay available inside the app.