Apache Spark Developer – Remote

Bright Vision Technologies is a technology consulting and software development company delivering cloud, AI, data, and enterprise solutions across the United States.

This is a fantastic opportunity to join an established and well-respected organization offering tremendous career growth potential.

Job Summary

We are seeking an experienced Apache Spark Developer to design, develop, and optimize large-scale distributed data processing applications supporting enterprise analytics, machine learning, real-time reporting, and cloud-based data platforms. This role focuses on building high-performance Spark applications capable of processing billions of records across structured and semi-structured data sources while delivering scalable, reliable, and cost-efficient data pipelines.

Key Responsibilities

• Design, develop, and maintain high-performance distributed data processing applications using Apache Spark.
• Build scalable batch and real-time ETL/ELT pipelines processing large volumes of enterprise data.
• Develop Spark applications using PySpark, Scala, or Spark SQL for data transformation, aggregation, and analytics.
• Optimize Spark jobs for memory utilization, partitioning strategies, shuffle performance, and execution efficiency.
• Process structured, semi-structured, and streaming data from enterprise databases, APIs, Kafka, cloud storage, and data lakes.
• Develop reusable Spark libraries, data processing frameworks, and metadata-driven ingestion pipelines.
• Collaborate with cloud engineering teams to deploy Spark workloads on Databricks, EMR, Azure Synapse, or Kubernetes.
• Implement data quality validation, reconciliation, monitoring, and automated error handling across distributed pipelines.
• Integrate Spark applications with enterprise data warehouses, lakehouses, and reporting platforms.
• Participate in architecture reviews, code reviews, technical design discussions, and Agile development activities.
• Troubleshoot production issues involving distributed processing, cluster performance, resource utilization, and data quality.
• Support cloud migration initiatives by modernizing legacy ETL workloads into Spark-based architectures.

Required Skills

• Six or more years of professional software or data engineering experience.
• Four or more years of hands-on Apache Spark development experience in enterprise production environments.
• Strong proficiency in PySpark, Scala, or Spark SQL for distributed data processing.
• Deep understanding of Apache Spark architecture including RDDs, DataFrames, Datasets, Catalyst Optimizer, DAG execution, and Tungsten engine.
• Strong experience with distributed computing concepts including partitioning, shuffling, caching, broadcast joins, and fault tolerance.
• Advanced SQL skills with databases such as SQL Server, Oracle, PostgreSQL, Snowflake, or Teradata.
• Experience working with Hadoop ecosystem technologies including Hive, HDFS, YARN, and Parquet.
• Experience processing streaming data using Spark Structured Streaming, Apache Kafka, or Event Hubs.
• Hands-on experience with cloud platforms including Azure Databricks, AWS EMR, AWS Glue, Azure Synapse Analytics, or Google Dataproc.
• Experience integrating Spark applications with Delta Lake, Apache Iceberg, or Apache Hudi.
• Strong understanding of data warehousing concepts, dimensional modeling, and data lake architecture.
• Experience using Git, CI/CD pipelines, Azure DevOps, GitHub Actions, or Jenkins.
• Strong debugging, troubleshooting, and Spark performance tuning skills.
• Experience working in Agile Scrum development environments.

Preferred Qualifications

• Experience building enterprise Lakehouse architectures using Databricks or Delta Lake.
• Familiarity with Apache Airflow, Azure Data Factory, AWS Step Functions, or Control-M for workflow orchestration.
• Experience with machine learning workflows using Spark MLlib, MLflow, or feature engineering pipelines.
• Knowledge of Kubernetes, Docker, and containerized Spark deployments.
• Experience implementing Data Quality frameworks using Great Expectations or Deequ.
• Familiarity with Apache NiFi, Apache Flink, Trino, or Presto.
• Experience working with cloud object storage including Amazon S3, Azure Data Lake Storage (ADLS Gen2), or Google Cloud Storage.
• Knowledge of Infrastructure as Code using Terraform or ARM templates.
• Experience with enterprise monitoring tools including Prometheus, Grafana, Datadog, or OpenTelemetry.
• Cloud certifications in Azure, AWS, Databricks, or Apache Spark-related technologies are highly desirable.

How to Apply

Would you like to know more about this opportunity? For immediate consideration, please send your resume to Harry@bvteck.com or contact us at (908) 676-4399. Learn more about Bright Vision Technologies at www.bvteck.com.