As a seasoned Data Engineer, you will play a pivotal role in designing, developing, and maintaining data pipelines that drive our threat research efforts forward. The primary focus of this position is to craft innovative solutions for ingesting threat intelligence data from diverse sources, ensuring seamless integration into our data ecosystem.
Key Responsibilities:
* Design and implement scalable data pipelines that cater to the ever-evolving needs of our threat research team.
* Develop and maintain robust data validation processes to guarantee accuracy, completeness, and consistency throughout our data sets.
* Collaborate closely with threat analysts to understand their requirements and design tailored solutions that meet their needs.
* Create automation scripts and workflows for efficient data export processes to external systems or partners.
* Enhance existing data pipelines to optimize performance, scalability, and reliability.
* Maintain vigilant monitoring of data pipelines, promptly identifying and resolving issues to ensure uninterrupted data availability and integrity.
* Document technical specifications, data flows, and procedures to facilitate effortless pipeline maintenance and support.
* Stay abreast of emerging technologies and best practices in data engineering, integrating them into our data ecosystem to stay ahead of the curve.
* Provide expert guidance and support to colleagues on data engineering best practices and methodologies.
Requirements and Qualifications:
* Proven experience as a Data Engineer or similar role, with a strong focus on data ingest, validation, and export automation.
* Demonstrated proficiency in Python programming language.
* Familiarity with data pipeline orchestration tools such as Apache Airflow, Apache NiFi, or similar platforms.
* Experience working with cloud-based platforms like Snowflake, AWS, Azure, or Google Cloud Platform.
* Strong understanding of data validation techniques and tools for ensuring data quality.
* Hands-on experience building and deploying containerized images using Docker and Kubernetes.
* Exceptional problem-solving skills, with a keen eye for detail.
* Excellent communication and collaboration skills, with the ability to work effectively within a team environment.
About the Role:
This is an exceptional opportunity for a highly skilled Data Engineer to join our team and contribute to the success of our threat research initiatives. If you are a motivated professional with a passion for data engineering and a commitment to excellence, we encourage you to apply for this exciting role.