Position Overview

The Commonwealth of Pennsylvania is seeking an experienced Data Engineer & Insights Analyst to support the Pennsylvania Longitudinal Data System (PALDS).

The selected candidate will design and maintain data pipelines, integrate and transform complex datasets, improve data quality, and develop analytics solutions to support program delivery, operational improvements, and executive decision-making across state government agencies.

This role requires strong technical expertise in AWS, SQL, Python, big data technologies, data modeling, and data integration, along with the ability to work with large, complex, and confidential datasets. The candidate will also support data conversions related to system modernization and replacement initiatives.

Required Skills

• 5+ years of hands-on experience with AWS cloud services, including S3, EC2, Glue, EMR, RDS, Redshift, Athena, Lambda, and DynamoDB.
• 5+ years of programming/scripting experience with Python, R, Java, C++, or Scala.
• 5+ years of experience with SQL and relational/NoSQL databases, including Oracle, SQL Server, PostgreSQL, and Cassandra.
• 5+ years of experience with Git, version control, and CI/CD pipelines.
• Strong knowledge of data modeling, including dimensional modeling, Data Vault, and star/snowflake schemas.
• Experience with AWS Glue Data Catalog, metadata management, and data cataloging.
• Understanding of data governance, data lineage, and data quality frameworks.
• Experience designing and integrating REST APIs.
• Hands-on experience with big data technologies such as Hadoop, Spark, and Kafka.
• Experience with data pipeline orchestration tools such as Apache Airflow, Azkaban, or Luigi.
• Experience with data integration platforms such as Informatica, MuleSoft, Talend, or TIBCO.
• Experience with stream-processing technologies such as Storm or Spark Streaming.
• Advanced SQL knowledge and experience working with large, complex datasets.
• Strong analytical, problem-solving, project management, and organizational skills.
• Experience supporting cross-functional teams in a dynamic environment.

Key Responsibilities

• Design, build, and maintain scalable data pipeline architectures.
• Develop cloud-based data solutions using AWS services such as S3, Glue, Redshift, EMR, Athena, Kinesis, and Lambda.
• Build and maintain data orchestration workflows using Apache Airflow, AWS Step Functions, or similar tools.
• Extract, transform, integrate, and load data from multiple sources using SQL and big data technologies.
• Develop and optimize data environments supporting statistical analysis, predictive modeling, machine learning, and research.
• Collaborate with data scientists, research analysts, and policy analysts to support reproducible research and data-driven decision-making.
• Identify and resolve data quality issues through validation, monitoring, and root-cause analysis.
• Automate manual processes and improve data delivery, infrastructure scalability, and operational efficiency.
• Develop dashboards, reports, analytical models, and decision-support tools.
• Build monitoring, alerting, and logging frameworks for data pipelines.
• Support data conversions associated with system modernization and replacement projects.
• Ensure compliance with data governance, privacy, security, and Commonwealth policies.
• Collaborate with executive, data, design, and support teams to address technical issues and improve data systems.

Preferred Skills

The job description does not specify a separate preferred-skills list. The technologies and experience listed under Required Qualifications and Required/Desired Skills should be treated as the stated requirements.

Primary Skills

AWS | Python | Java | R | SQL | Oracle | SQL Server | PostgreSQL | Cassandra | Data Modeling | AWS Glue | S3 | EC2 | EMR | RDS | Redshift | Athena | Lambda | DynamoDB | Git | CI/CD | REST APIs | Hadoop | Spark | Kafka | Airflow | Azkaban | Luigi | Informatica | MuleSoft | Talend | TIBCO | Storm | Spark Streaming

This is a remote position.