Senior big data engineer

Our client is seeking an experienced Big Data Engineer to design, develop, and support large-scale data pipelines and transformation processes. The ideal candidate will have strong SQL, Python, ETL/ELT, and data engineering experience, along with the ability to communicate technical concepts clearly to clients, stakeholders, and cross-functional teams.

Java experience is helpful for supporting existing systems and integrations but is not the primary focus of this position.

Responsibilities

• Design, build, maintain, and optimize ETL/ELT pipelines that move and transform large-scale data across source and target systems.
• Write and optimize complex SQL queries, stored procedures, data models, and validation processes.
• Develop data transformation, integration, and automation solutions primarily using Python.
• Use Java as needed to support existing applications, services, and system integrations.
• Parse, transform, and validate XML-based data formats.
• Work within Linux and shell environments to support scripting, scheduled jobs, cron processing, log reviews, and troubleshooting.
• Implement data validation, error handling, monitoring, and traceability controls.
• Troubleshoot pipeline failures, data discrepancies, performance issues, and integration defects.
• Collaborate with client, security, infrastructure, data engineering, and delivery teams.
• Communicate technical issues, risks, and recommendations clearly to technical and nontechnical stakeholders.
• Participate in Agile planning, status reporting, documentation, issue tracking, and cross-team coordination.

Required Qualifications

• Bachelor’s degree from an accredited college or university or equivalent professional experience.
• Five or more years of professional experience in data engineering, database development, ETL/ELT development, or a related data integration role.
• Strong hands-on SQL experience, including complex queries, performance tuning, stored procedures, and data modeling.
• Experience designing, building, optimizing, and supporting large-scale data pipelines.
• Strong Python programming experience supporting data transformation, integration, automation, or pipeline operations.
• Experience parsing, transforming, and validating XML-based data.
• Experience working in Linux and shell environments, including scripting, scheduled jobs, cron, and log troubleshooting.
• Experience implementing data validation, error handling, pipeline monitoring, and production troubleshooting.
• Strong written and verbal communication skills.
• Ability to explain technical concepts clearly to clients and stakeholders.
• Ability to collaborate across security, infrastructure, engineering, and delivery teams.
• Self-motivated and proactive, with a positive and collaborative approach.
• U.S. citizenship is required.
• Must be willing and able to undergo a background investigation for a Public Trust suitability determination.

Preferred Qualifications

• Experience using Python as the primary language for data pipeline automation, transformation, monitoring, or production troubleshooting.
• Experience supporting Java-based applications, services, or integrations.
• Working knowledge of AWS data services, including S3, Glue, Lambda, RDS, or Redshift.
• Exposure to XBRL or other structured financial, regulatory, or compliance-driven data formats.
• Experience supporting federal, government, or compliance-driven data environments.
• Experience with Oracle, PostgreSQL, Redshift, or other relational database platforms.
• Familiarity with reporting, analytics, or dashboarding tools such as Power BI or MicroStrategy.