Data Engineer
Do we have a match?
As a Data Engineer, you must meet the following criteria:
- Strong Python experience: data structures, typing, error handling, generators/iterators, context managers, and the standard library
- Software engineering fundamentals: modular design, dependency management, packaging, and API design
- Testing discipline: pytest, fixtures, mocking, and writing code that's testable by construction
- Hands-on experience building and operating ETL/ELT pipelines in production, not just scripts or notebooks
- Strong experience with Snowflake and dbt
- Experience with Apache Airflow or similar code-based orchestration tools
- Solid working knowledge of SQL and data modeling, sufficient to design robust database schemas and query them effectively
- Experience with Docker, Kubernetes, and CI/CD practices
- Debugging and profiling skills; able to reason about performance, concurrency, and memory in Python
- Experience with at least one public cloud (AWS or Azure)
- Experience with version control systems (Git)
- Able to explain technical trade-offs to both technical and business audiences
- Experience using AI coding assistants such as Claude Code, Cursor, or similar on a daily basis
- Strong communication skills and good knowledge of English (minimum C1 level)
Beyond the criteria above, we would appreciate the following nice-to-haves:
- Experience with Apache Spark, ideally on Databricks
- Experience with Pydantic or similar schema-validation libraries
- Python web/API frameworks (FastAPI, Flask)
- Async Python, multiprocessing, or other concurrency patterns
- Experience with Azure AI Search or AWS OpenSearch
- A second language: Go, Rust, Scala, or TypeScript
- Familiarity with LLMs, Azure OpenAI, or agentic AI systems
The project and the team
You will join the team behind a large-scale, centralized data platform built for a global consulting organization. The platform is the shared source of company data behind several of the firm's internal products, and is used daily by consultants for company research, including support for Mergers & Acquisitions (M&A) engagements.
This is fundamentally a data engineering role combined with software engineering: you'll be designing, coding, testing, and operating production Python systems - pipelines, libraries, and services - that move, transform, and serve data at scale. This is not a role focused on configuring tools or writing one-off queries. You'll be building reliable, maintainable software that powers our data platform. The goal is a unified, enterprise-grade dataset of 300M+ company records, integrated from 10+ external and internal sources.
The platform delivers firm-level and site-level data - firmographics, technographics, and hierarchical relationships (parent company, subsidiary, site) - alongside key business metrics such as revenue, CAGR, EBITDA, headcount, M&A activity, competitors, industry classification, and web traffic. Data needs to stay accurate, well-structured, and fast to query as both the dataset and the number of consumers keep growing.
Technology stack:
- Languages: Python, SQL
- Data platform: Snowflake, dbt
- Workflow orchestration: Apache Airflow (complex DAGs), running on Kubernetes
- Data processing: Apache Spark on Azure Databricks
- Data tooling and DBs: pandas, Polars, PyArrow, DuckDB, PySpark, PostgreSQL, Redis
- Cloud: Azure (AKS, Blob Storage, ACR, Databricks, OpenSearch, Azure AI Search)
- API & services: FastAPI (REST, async), API Gateway
- Testing & code quality: pytest, mypy/pyright, ruff/black, sqlfluff, SonarQube
- Schema validation: Pydantic
- Dependency & environment management: uv, Poetry
- CI/CD & infrastructure: GitHub Actions, Docker, Kubernetes
- AI-Assisted Development: Cursor, Claude Code, ChatGPT Enterprise
- Future direction: agentic AI systems, LangChain, Azure OpenAI integration
What else you should know:
- Team: Data Architecture Lead, Data Engineers, DataOps Engineers, Backend Engineer, Product Owner, collaboration with Frontend Engineers and Data Science and AI Engineers
- Distributed team across Europe and India
- Agile, collaborative environment; given the platform's organization-wide impact, we're looking for a mature, proactive, results-driven approach
- Code quality is enforced through testing, typing, and tooling - not just code review
We work on multiple interesting projects at a time, so it may happen that we’ll invite you to an interview for another project if we see that your competencies and profile are well suited for it.
More reasons to join us
- Flexible working hours and approach to work: fully remotely, in the office or hybrid
- Professional growth supported by internal training sessions and a training budget
- Solid onboarding with a hands-on approach to give you an easy start
- A great atmosphere among professionals who are passionate about their work
- The ability to change the project you work on