What is the Position?
Evercore seeks to hire a Director, Lead Engineer to serve as the technical authority for data engineering standards and platform architecture within Global Technology’s AI, Data & Analytics organization. This role is responsible for defining and governing how data pipelines, data products, and AI workloads are architected, built, secured, and operated on the firm’s Databricks platform.

This high-impact role sits at the intersection of platform engineering, distributed systems design, and developer experience. The ideal candidate will bring deep expertise in Databricks, Spark, enterprise data governance and event-driven architectures for regulated environments.

Primary Responsibilities:
We are seeking an experienced technical leader with deep Databricks expertise to define strategy and establish architectural standards across the platform. This role will partner closely with business, technology, security, and infrastructure leaders to ensure the platform is scalable, compliant, cost-efficient, and aligned to long-term enterprise objectives. Responsibilities include, but are not limited to the following:

• Define and govern enterprise architecture standards for batch and streaming data pipelines on Azure Databricks, including Spark optimization, structured streaming, and pub/sub integration (e.g., Event Hub, Kafka).
• Design and maintain standardized internal SDKs and reusable frameworks for ETL development.
• Establish the data engineering SDLC, including schema evolution policies, artifact versioning, environment promotion controls and CI/CD standards.
• Design secure storage architectures leveraging Azure Data Lake Storage Gen2 (ADLS Gen2).
• Architect and administer secure, scalable Databricks environments, including workspace segmentation strategies, Unity Catalog design, role-based and attribute-based access controls, row/column-level security, and compute isolation policies.
• Design and implement enterprise metadata, observability, and data quality frameworks, including automated lineage capture, SLA monitoring, operational telemetry, and cost attribution models.
• Establish data product standards, including schema contract enforcement, versioning policies, backward compatibility requirements, domain ownership models, and publish/subscribe consumption patterns.
• Define best practices for exploratory and non-production workloads, including guardrails for data scientist experimentation, compute policy controls, secure data access boundaries, and promotion of validated assets into production pipelines.
• Partner with Data Scientists and Machine Learning Engineers to define the lifecycle from experimentation to production, including feature engineering standards, model registry governance, reproducibility controls, and approved deployment pathways.

Specific Qualifications:

• Minimum 10+ years of experience in data engineering, enterprise data platform architecture, or large-scale analytics platforms.
• Minimum 5+ years of hands-on experience designing and scaling Databricks and Spark-based platforms in enterprise environments.
• Deep expertise in Apache Spark (performance tuning, optimization strategies, structured streaming, distributed compute patterns).
• Experience architecting Azure cloud infrastructure.
• Proven experience architecting event-driven and streaming data systems using pub/sub technologies such as Kafka or equivalent platforms.
• Demonstrated experience designing enterprise-grade ETL/ELT frameworks and internal SDKs for standardized pipeline development and deployment.
• Strong experience implementing Unity Catalog or equivalent enterprise data governance tooling, including fine-grained access control models and regulatory audit readiness.
• Experience operating within regulated or highly controlled environments with formal compliance and audit requirements.
• Bachelor’s degree in Computer Science, Engineering, or related quantitative discipline required; advanced degree preferred.
• This position requires valid United States work authorization and is not eligible for employer visa sponsorship.

Expected Base Salary Range: $210,000 - $230,000  

In addition to a competitive base salary, employees may be eligible to receive a discretionary bonus delivered in the form of cash and/or deferred equity. Evercore also offers a variety of benefits and programs, subject to eligibility. These include, but are not limited to:

• Medical, prescription, dental, and vision insurance, including healthcare savings and reimbursements accounts
• 401(k) Retirement Plan  
• Life and disability insurance, including additional voluntary financial protection insurance
• Well-being resources and programs, including mental health and mindfulness programs, digital wellness platforms, well-being events, and targeted on-site health services
• Family-building and family-support benefits
• Paid parental, caregiver, marriage and bereavement leave
• Commuter benefits, health club membership discounts, and other corporate discounts
• Paid holidays, vacation days, personal days, sick days, and volunteer opportunities