Job Title
We are seeking a skilled and experienced Data Engineer to join our team.
Responsibilities
* Design, develop, and maintain data pipelines for ingesting threat intelligence data from various sources into our ecosystem.
* Implement data validation processes to ensure data accuracy, completeness, and consistency.
* Collaborate with threat analysts to understand data requirements and design appropriate solutions.
* Develop automation scripts and workflows for data export processes to external systems or partners.
* Optimize and enhance existing data pipelines for improved performance and scalability.
* Monitor data pipelines and troubleshoot issues as they arise, ensuring continuous data availability and integrity.
* Document technical specifications, data flows, and procedures for data pipeline maintenance and support.
* Stay updated on emerging technologies and best practices in data engineering and incorporate them into our ecosystem.
* Provide technical guidance and support to other team members on data engineering best practices and methodologies.
Requirements
* Proven experience as a Data Engineer or similar role, with a focus on data ingest, validation, and export automation.
* Strong proficiency in Python.
* Experience with data pipeline orchestration tools such as Apache Airflow, Apache NiFi, or similar.
* Familiarity with cloud platforms such as Snowflake, AWS, Azure, or Google Cloud Platform.
* Experience with data validation techniques and tools for ensuring data quality.
* Experience building and deploying images using containerization technologies such as Docker and Kubernetes.
* Excellent problem-solving skills and attention to detail.
* Strong communication and collaboration skills, with the ability to work effectively in a team environment.