Cloud Data Solutions Engineer
Job Overview:
As a Cloud Data Solutions Engineer, you will be responsible for designing and developing scalable data processing pipelines using cloud-based services.
You will collaborate with data scientists and analysts to understand data requirements and deliver high-quality datasets.
Optimizing data workflows for performance and cost efficiency in a cloud environment is also a key part of this role.
Additionally, you will implement best practices around data security, governance, and compliance.
Key Responsibilities:
Design and develop ETL/ELT pipelines using Azure Data Services.
Build high-performance data processing solutions using Apache Spark on Azure Databricks.
Collaborate with data scientists, analysts, and stakeholders to understand data requirements.
Optimize data workflows/pipelines for performance and cost efficiency in a cloud environment.
Implement best practices around data security, governance, and compliance.
Develop CI/CD pipelines for data engineering workflows.
Requirements:
Experience with columnar and table formats (Parquet, Delta, Hudi, Iceberg).
Use of Data Quality tools (Great Expectations, Soda).
Knowledge of Step Functions, EventBridge, or Kinesis.
Best practices for API security (Cognito, WAF, IAM Policies).
Soft Skills: Good communication and teamwork, proactivity in problem-solving, ability to handle agile environments and rapid changes.
What We Offer:
Professional development and constant evolution of your skills.
Opportunities to work in a collaborative, diverse and innovative environment that encourages teamwork.
About Us:
Tata Consultancy Services promotes an inclusive culture and values diversity, equity, and inclusion in all its opportunities.