Cloud Data Engineer
This role involves designing, developing, and maintaining large-scale data pipelines using Azure services.
-----------------------------------
Responsibilities:
* Data Pipeline Development: Design and develop efficient ETL/ELT pipelines for data processing, utilizing Azure Data Services to ensure scalability and reliability.
* Big Data Processing: Develop high-performance data processing solutions with Apache Spark (PySpark) on Azure Databricks to handle large datasets efficiently.
* Collaboration: Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver accurate, reliable datasets.
* Workflow Optimization: Optimize data workflows/pipelines for performance and cost efficiency in a cloud environment, ensuring seamless integration with existing systems.
* Security and Compliance: Implement best practices around data security, governance, and compliance to protect sensitive information.
* CICD Pipelines: Develop CI/CD pipelines for data engineering workflows to automate testing, deployment, and monitoring.
* Solution Enhancement: Monitor, troubleshoot, and enhance existing data solutions for reliability, performance, and scalability.
* Documentation: Document design patterns, best practices, and operational procedures to facilitate knowledge sharing and collaboration.
Requirements:
* Experience with columnar and table formats (Parquet, Delta, Hudi, Iceberg).
* Use of Data Quality tools (Great Expectations, Soda).
* Knowledge of Step Functions, EventBridge, or Kinesis.
* Best practices for API security (Cognito, WAF, IAM Policies).
* Soft Skills: good communication and teamwork, proactivity in problem-solving, ability to handle agile environments and rapid changes.
About the Role:
This is an exciting opportunity to join our team of experienced professionals in cloud data engineering. As a Cloud Data Engineer, you will play a key role in designing and implementing scalable data solutions that drive business growth and innovation.
We offer a dynamic work environment that fosters collaboration, creativity, and continuous learning. Our team is passionate about delivering high-quality solutions that meet the evolving needs of our clients and stakeholders.