DescriptionHelp modernize cloud data engineering by delivering trusted lakehouse products and shaping responsible AI-assisted engineering practices that improve engineering throughput and quality.
Job summary
As a Software Engineer III at JPMorgan Chase within the Cloud Data Engineering team, you design, build, and operate scalable data engineering products on AWS using Databricks and lakehouse architecture. You use AI-assisted delivery practices (for example, GitHub Copilot and Claude) to accelerate delivery across development, testing, documentation, and production support, while we maintain human accountability for architecture, approvals, security, and production decisions.
Job responsibilities
- Design scalable extract-transform-load (ETL) and extract-load-transform (ELT) pipelines on AWS using Databricks, PySpark, Spark, Delta Lake, AWS Glue, SQL, and lakehouse patterns for batch and streaming workloads.
- Build ingestion, transformation, enrichment, calculation, allocation, reporting, and publishing pipelines that produce maintainable, well-modeled data products.
- Develop production-grade Python, PySpark, and SQL with disciplined engineering practices including testing, peer review, performance tuning, resiliency, and secure coding.
- Operate orchestration and scheduling workflows using Airflow (or equivalent), including dependency management, retries, backfills, service level agreement monitoring, and operational runbooks.
- Implement continuous integration and continuous delivery practices with automated quality gates (linting, unit and integration tests, security scans, data-quality checks, schema validation, and release controls).
- Build and maintain infrastructure as code using Terraform and/or AWS CloudFormation, and create architecture, design, lineage, and operational artifacts that meet scalability, performance, cost, resiliency, governance, and security requirements.
- Lead adoption of AI-assisted engineering practices across the ETL lifecycle (Copilot-assisted development, Claude-enabled refactoring and troubleshooting, and reusable AI skills and task-oriented agents) and integrate approved developer tools with Jira, source control, documentation, observability, incident triage, and release readiness—ensuring controls for human review, secrets protection, and safeguards for personally identifiable information and payment card information.
Required qualifications, capabilities, and skills
- 8 years of hands-on data engineering or software engineering experience delivering production data platforms and pipelines.
- Hands-on experience building data products using Databricks, Spark, Delta Lake, and lakehouse architectures on AWS.
- Advanced proficiency in Python, PySpark, and SQL, with demonstrated ability to write clean, testable, maintainable, production-grade code.
- Hands-on experience with AWS Glue and cloud-native ETL patterns, including jobs, cataloging, identity and access management security, monitoring, and operational support.
- Experience with Airflow (or equivalent orchestration), including workflow design, scheduling, backfills, dependency management, monitoring, and recovery.
- Hands-on Apache NiFi experience for ingestion design, flow operationalization, source onboarding, and reliability practices.
- Demonstrated experience with disciplined delivery practices, including infrastructure as code (Terraform and/or AWS CloudFormation), automated testing and quality gates, release controls, agile delivery, DevOps practices, data governance, resiliency engineering, secure-by-design implementation, AI-assisted workflows (including GitHub Copilot and Claude) with human review, reusable AI skills/prompts/agents, and controls for sensitive data (personally identifiable information and payment card information), secrets management, access control, auditability, and secure logging.
Preferred qualifications, capabilities, and skills
- Experience implementing agentic automation patterns for incident triage, deployment validation, data-quality checks, lineage generation, developer enablement, or operational workflows.
- Experience integrating AI agents with Jira, source control systems, documentation platforms, Databricks, observability platforms, or internal developer tooling.
- Familiarity with data mesh concepts, Unity Catalog, data contracts, metadata management, and modern analytics and business intelligence ecosystems (for example, ThoughtSpot), including streaming and event-driven ingestion (for example, Kafka) and enterprise observability tooling.
- AWS and/or Databricks certifications (for example, AWS Solutions Architect, AWS Developer Associate, AWS Data Analytics Specialty, or Databricks Data Engineer certification) and experience mentoring engineers and leading standards for AI-assisted data engineering delivery.
