Job Title

Lead Data Engineer

Company Profile

Company Description

  • SPOTYOURJOB Premium subscription is unique. We provide Priority access to newly listed jobs Resume & Cover letter writing service LinkedIn page makeover Mock interviews and career guidance by experts We search & apply matching job for you Overseas career guidance and assistance"

Job Description

  • Responsibilities:


    ·        Leveraging your deep knowledge to provide technical leadership to take projects from zero to completion.
    ·        Architect, build and maintain scalable data pipelines and access patterns related to permissions and security.
    ·        Research, evaluate and utilize new technologies/tools/frameworks centered around high-volume data processing
    ·        Drive a culture of collaborative reviews of designs, code and test plans
    ·        Work with the architecture engineering team to ensure quality solutions are implemented and engineering best practices adhered to
    ·        Develop set process for Data mining, Data modeling, and Data production

    Basic Requirements:

    ·        A data engineer with strong programming experience in Python
    ·        Extensive experience related to processing frameworks such as Spark, Spark Streaming, Airflow, Hive, Sqoop, Kafka etc.
    ·        Experience with big data processing within cloud environments such AWS S3.
    ·        Passion for Data Privacy and Security, Data Management.
    ·        Experience building and shipping data production pipelines sourcing data from a diverse array of sources.
    ·        Deep understanding of measuring and ensuring data quality at scale and the required tooling to monitor and optimize the performance of our data pipelines
    ·        Ability to influence and communicate effectively with team members and business stakeholders

    Nice to have:
    ·        Knowledge of the retail and e-commerce sector and its use cases.

    Key Skills:
    ·       Scala

    .       Oracle/MySQL
    ·       Airflow
    ·       Spark
    ·       AWS Cloud (EMR, S3, SNS, SQS)
    .       S3 Data Lake  

    .       Kafka

  • Job Type


  • Designation

    Lead Data Engineer

  • Stream

    Data Analytics ( Bigdata.. )

  • Location

    Bangalore - India

  • Experience

    7.00 - 15.00 yrs

  • No of Positions


  • Mandatory Skills


  • Desired Skills