Pracuj.pl
Hybrydowo
Senior
New
Senior Data Engineer (AWS)
⚲ Gdańsk, Oliwa, Katowice, Kraków, Grzegórzki, Lublin, Opole, Poznań, Grunwald, Warszawa, Włochy, Wrocław, Fabryczna
Do uzgodnienia
Wymagania
- Apache Spark
- Microsoft Azure
- Python
Opis stanowiska
Nasze wymagania:
Microsoft Fabric or Databricks production engineering
Apache Spark and Spark UI troubleshooting
Shuffle skew, spill, OOM, task-duration analysis, AQE, join strategy, and predicate pushdown
Delta Lake optimization: liquid clustering, partitioning, Z-order, file sizing, OPTIMIZE, statistics, VACUUM, and incremental MERGE
Fabric capacity economics: CU-seconds, bursting, smoothing, throttling stages, pool configuration, node sizing, concurrency, and workload isolation
Production Python library development, packaging, validation, logging, testing, versioning, and documentation
Operational pipeline design: idempotency, watermarks, quarantine, monitoring, failure recovery, and SLA awareness
O projekcie:
Our client is expanding its data platform team as part of its ongoing data modernization initiatives. They are looking to add Data Engineer to support the evolution of the Microsoft-based analytics and data platform.
Empoloyee will need to be able to provide overlap with the customer's US Eastern Time working hours, including availability through approximately 4:30 PM ET.
Zakres obowiązków:
Design, optimize, and operate reliable, large-scale, cost-aware Spark and Delta Lake workloads.
Build and optimize Spark pipelines on Microsoft Fabric or Databricks. Diagnose performance using Spark UI and Delta statistics. Manage Delta Lake design, Fabric capacity consumption, compute sizing, throttling exposure, and operational reliability. Create reusable Python libraries and Spark modules for ingestion, governance, security, monitoring, and optimization.
Demonstrated production troubleshooting and capacity-economics experience, not only notebook or ETL development.
Measures workload behavior before changing infrastructure; optimizes code and configuration before increasing compute; documents pool sizing, throttling risk, and operational requirements; explains capacity behavior to platform and engineering teams.
Oferujemy:
Well-being culture: medical care with Medicover, private life insurance, and sports card. But we went one step further by creating our own Capgemini Helpline offering therapeutical support if needed and the educational podcast "Let's talk about wellbeing" which you can listen to on Spotify.
Access to over 70 training tracks with certification opportunities (e.g., GenAI, Architects, Google) on our NEXT platform. Dive into a world of knowledge with free access to Education First languages platform, TED Talks and Udemy Business materials and trainings.
Continuous feedback and ongoing performance discussions thanks to our performance management tool GetSuccess supported by a transparent performance management policy.
Enjoy hybrid working modelonboarding that fits your life - after completing , connect work from a modern office with ergonomic work from home, thanks to home office package (including laptop, monitor, and chair). Ask your recruiter about the details.
Microsoft Fabric or Databricks production engineering
Apache Spark and Spark UI troubleshooting
Shuffle skew, spill, OOM, task-duration analysis, AQE, join strategy, and predicate pushdown
Delta Lake optimization: liquid clustering, partitioning, Z-order, file sizing, OPTIMIZE, statistics, VACUUM, and incremental MERGE
Fabric capacity economics: CU-seconds, bursting, smoothing, throttling stages, pool configuration, node sizing, concurrency, and workload isolation
Production Python library development, packaging, validation, logging, testing, versioning, and documentation
Operational pipeline design: idempotency, watermarks, quarantine, monitoring, failure recovery, and SLA awareness
O projekcie:
Our client is expanding its data platform team as part of its ongoing data modernization initiatives. They are looking to add Data Engineer to support the evolution of the Microsoft-based analytics and data platform.
Empoloyee will need to be able to provide overlap with the customer's US Eastern Time working hours, including availability through approximately 4:30 PM ET.
Zakres obowiązków:
Design, optimize, and operate reliable, large-scale, cost-aware Spark and Delta Lake workloads.
Build and optimize Spark pipelines on Microsoft Fabric or Databricks. Diagnose performance using Spark UI and Delta statistics. Manage Delta Lake design, Fabric capacity consumption, compute sizing, throttling exposure, and operational reliability. Create reusable Python libraries and Spark modules for ingestion, governance, security, monitoring, and optimization.
Demonstrated production troubleshooting and capacity-economics experience, not only notebook or ETL development.
Measures workload behavior before changing infrastructure; optimizes code and configuration before increasing compute; documents pool sizing, throttling risk, and operational requirements; explains capacity behavior to platform and engineering teams.
Oferujemy:
Well-being culture: medical care with Medicover, private life insurance, and sports card. But we went one step further by creating our own Capgemini Helpline offering therapeutical support if needed and the educational podcast "Let's talk about wellbeing" which you can listen to on Spotify.
Access to over 70 training tracks with certification opportunities (e.g., GenAI, Architects, Google) on our NEXT platform. Dive into a world of knowledge with free access to Education First languages platform, TED Talks and Udemy Business materials and trainings.
Continuous feedback and ongoing performance discussions thanks to our performance management tool GetSuccess supported by a transparent performance management policy.
Enjoy hybrid working modelonboarding that fits your life - after completing , connect work from a modern office with ergonomic work from home, thanks to home office package (including laptop, monitor, and chair). Ask your recruiter about the details.
🔍 Dekoder Ogłoszenia
✓ Ogłoszenie wygląda transparentnie — brak typowych czerwonych flag.