Lead Software Engineer - Databricks, Spark, AWS
at J.P. Morgan
Posted 17 hours ago
No clicks
- Compensation
- Not specified USD
- City
- Not specified
- Country
- United States
Currency: $ (USD)
Lead Software Engineer responsible for architecting and delivering high-throughput, low-latency data pipelines using Databricks and Apache Spark, with a focus on lakehouse patterns and scalable data platforms. Own Databricks cluster strategy, job orchestration with Databricks Workflows, and integration with AWS services (S3, Glue, IAM, CloudWatch, Kinesis/MSK) for secure data ingestion and processing. Develop reusable libraries in Python/Java, implement CI/CD for data projects, enforce data governance and security, and drive reliability, observability, and cost optimization across production pipelines. Lead multiple teams in a fast-paced CTO environment at JPMorgan Chase, shaping data architectures and ensuring SLAs and readiness for production.
Location: Plano, TX, United States
This is your chance to change the path of your career and guide multiple teams to success at one of the world's leading financial institutions.
As a Lead Software Engineer at JPMorgan Chase within Corporate Sector, Chief Technology Office, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm’s business objectives.
Job Responsibilities:
- Lead architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark (Core, SQL, Structured Streaming).
- Establish lakehouse patterns with Delta Lake (ACID transactions, schema evolution, time travel, Z-ordering, compaction) and ensure performance at scale.
- Own Databricks cluster strategy and setup: runtime selection, autoscaling, driver/executor sizing, Spark configs, init scripts, cluster policies, pools, and instance profiles.
- Orchestrate jobs with Databricks Workflows; integrate with AWS eventing and orchestration as needed.
- Design secure data ingestion and transformation frameworks leveraging AWS services:
- S3 for data lake storage and lifecycle management
- Glue for catalog/metadata and ETL jobs
- IAM and Secrets Manager for role-based access and credential management
- CloudWatch for logging, metrics, and alerting
- Lambda for serverless utilities
- Kinesis and/or Kafka/MSK for streaming ingestion
- Enforce data quality, lineage, and governance using Unity Catalog and/or Glue Catalog; embed expectations and validation into pipelines.
- Drive Spark performance engineering: partitioning strategies, file sizing, AQE, broadcast joins, shuffle tuning, caching, spill/memory control, and job right-sizing to optimize cost.
- Build reusable libraries, frameworks, and APIs in Python and/or Java; oversee unit, integration, and data validation testing.
- Implement CI/CD for data projects (Git-based workflows), Terraform Infrastructure deployments environment promotion, and automated deployments; champion engineering standards and code reviews.
- Partner with platform security and networking teams to enforce encryption, network controls, and least-privilege access; ensure compliance with organizational policies.
- Lead incident response and root-cause analysis; establish SLAs, observability, and runbooks; drive continuous improvement in reliability and cost efficiency.
- Formal training or certification on software engineering concepts and 5+ years applied experience.
- 10+ years of professional software/data engineering experience, including substantial production work with Spark on Databricks or EMR.
- Strong proficiency in Python and/or Java for data processing, platform tooling, and automation.
- Hands-on Databricks expertise (Delta Lake, Unity Catalog, Workflows, Repos/notebooks, SQL Warehouses).
- Solid AWS experience: S3, IAM, Glue, CloudWatch, Kinesis / MSK, DynamoDB
- Proven track record architecting and operating ETL/ELT pipelines (batch and streaming), with schema design/evolution, SLAs, and reliability engineering.
- Deep skills in Spark performance tuning and Databricks cluster setup/optimization.
- Strong SQL and analytics data modeling (dimensional/star schema; lakehouse best practices).
- Security-first mindset: roles/instance profiles, secret management, encryption-at-rest/in-transit, and network controls.
- Demonstrated leadership in code quality, reviews, testing strategy, CI/CD, and technical mentorship; excellent communication with stakeholders.
- Experience with Delta Live Tables and advanced governance (catalogs, grants, auditing) in Databricks.
- AWS networking knowledge (VPC, subnets, routing, security groups) and data egress controls.
- Experience with Terraform for Infra deployments
- Cost optimization experience: autoscaling strategies, spot vs on-demand, auto-termination, storage layouts and compaction.
- Familiarity with Kafka/MSK or Kinesis Data Streams/Firehose for real-time ingestion.
- CI/CD and automation tooling for data (Git workflows, artifact management) and testing frameworks (pytest, JUnit).
- Observability for data systems (freshness/completeness metrics, lineage, SLAs, alerting).
- Experience in financial services or other regulated industries.
