Responsibilities: Design, develop and build real-time / batch data pipelines from a variety of sources (streaming data, APIs, data warehouse, messages etc.) Leverage the understanding of software architecture and software design patterns to write scalable, maintainable, well-designed and future-proof software Manage existing pipelines and create new pipelines from a variety of sources (relational, XML, etc.) Actively apply best practices within CI/CD Propose and implement solutions for data pipeline stabilization and data quality checks Coordination with other teams to design optimal patterns for data ingest and egress, as well as lead and coordinate data quality initiatives and troubleshooting Design and build solutions to track data quality, stabilize data pipeline, etc. to ensure reliable operations Ensure best practices are followed across architecture, codebase and configuration Eliminate waste Deliver on time
Competences:
• Ability to establish with clear goals and responsibilities to achieve a high level of performance.
• Ability to evaluate different options proactively and ability to solve problems in an innovative way. Develop new solutions or combine existing methods to create new approaches.
• Comfortable in working with external product teams to establish the optimal data integration patterns/solutions
Functional Knowledge:
Azure based requirements:
1. Familiar with Azure storage account, Databricks, AD group, Key vault
2. Familiar with Azure DevOps pipeline, yaml configuration.
3. Familiar with Spark, know how to configure, customize spark, write pyspark code
4. Understand Mlflow, DBFS in Databricks
GCP requirements:
1. Familiar with BigQuery, can code SQL
2. Familiar with Cloud composer / airflow
3. Familiar with IAM, service account
4. Familiar with Data catalog
5. Understand Infrastructure as Code
6. Good to have knowledge with Dataflow, K8s, Vertex AI pipeline, Kubeflow pipeline
Cloud Agnostic Skills:
Python
• Deep knowledge about python programing, practice OOP, following coding best practice, know how to use flake8, mypy, black, SonarQube and pre-commit
• Deep knowledge in unit test and end to end test, familiar with Pytest, fixtures, unittest etc
Unix
• Familiar with popular Unix system, know how to install sth in docker.
• Familiar with shell
Git
• Know how to create PR and solve merge conflict.
• Can create CI/CD pipeline in either Github Action or Azure DevOps using best practice
Docker
• Deep understanding with Docker
DBT
• Deep Knowledge in DBT, preferably with GCP
SQL
• Deep knowledge of SQL
• Deep understanding with Data modeling, system design
Required cloud certification:
GCP ML Engineer -or- GCP Data Engineer
Required skills:
GCP certificate: ML Engineer or Data Engineer
Docker
SQL
Continuous Delivery (CD)
Azure
Azure DevOps
Machine Learning (ML)
Continuous Integration (CI)
Python
Google Cloud Platform (GCP)
Cubane is a Swedish IT consulting firm with over ten years of experience and a proven track record of assembling and leading teams of qualified IT specialists. Cubane is a business that is managed by a group of strong female executives. Read More
