Senior Data Engineer

Role Summary
NTT DATA is seeking a highly experienced Senior Data Engineer / Technical Lead for a large-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg.
The role requires a strong hands-on engineer who can lead the engineering team while continuing to design, develop, troubleshoot and optimize the solution. The candidate will own technical execution of the migration framework, Spark/PySpark engineering, Iceberg implementation, data modelling/schema mapping, Cyber/TaaS integration, migration controls, reconciliation, performance tuning and technical delivery of migration waves.
This is not a coordination-only Technical Lead role. The candidate is expected to contribute to code, conduct design/code reviews, troubleshoot complex production issues and mentor Data Engineers.
Experience
10+ years of Data Engineering experience, including:
5+ years hands-on Apache Spark/PySpark.
Strong Hadoop/Hive experience.
Experience leading Data Engineering teams.
Large-scale data migration/data modernization experience.
Strong data modelling and source-to-target mapping experience.
Multi-hundred-TB/PB-scale migration experience is strongly preferred.
Key Responsibilities
Technical Leadership
Lead a team of Data Engineers through assessment, development, migration, validation and production execution.
Translate architecture and migration standards into detailed engineering designs and reusable implementation patterns.
Own technical estimates, engineering backlog, work allocation and technical delivery.
Conduct design and code reviews and enforce engineering standards.
Mentor Data Engineers and provide hands-on technical guidance.
Act as the primary engineering escalation point for complex migration and production issues.
Partner with Data Architect, Technical PM, security, infrastructure and PNC technical teams.
Lead root-cause analysis and ensure fixes are incorporated into reusable framework patterns rather than implemented as one-off solutions.
Hadoop-to-Iceberg Migration
Lead technical execution of approximately 2 PB of Hadoop/Hive historical-data migration to Apache Iceberg.
Harden and extend the reusable, configuration-driven migration framework developed during the POC.
Analyze Hive/HDFS metadata, tables, partitions, file layouts and migration complexity.
Implement migration strategies including bulk/CTAS, partition-wise and chunk-wise processing.
Design execution approaches for small, medium, large and very-large tables.
Optimize parallel migration across tables and partitions.
Implement checkpointing, restart/recovery, idempotency and safe retries.
Lead technical execution and troubleshooting of migration waves.
Data Modelling & Schema Engineering
Analyze existing Hive data models and translate them into approved Apache Iceberg physical models.
Develop and review conceptual/logical/physical data models where required, with primary emphasis on physical modelling for migration.
Perform source-to-target data mapping and schema/data-type mapping.
Define and implement table structures consistent with approved Iceberg standards.
Analyze:
o entities and relationships,
o primary/foreign-key relationships,
o data types,
o partition keys,
o business keys,
o reference/cross-reference structures.
Identify schema incompatibilities and recommend remediation.
Work with the Data Architect on partitioning, file-layout and model-design decisions.
Ensure migrated models preserve required business relationships and downstream compatibility.
Support future Bronze/Silver/Gold modelling requirements where applicable.
Important: enterprise modelling standards remain owned by the Data Architect; the Senior DE/Technical Lead is responsible for detailed modelling and implementation within those standards.
Spark/PySpark Engineering & Performance
Develop and review production-grade Python/PySpark components.
Tune Spark jobs for PB-scale migration.
Diagnose Spark stages, executors, shuffle, skew, memory and I/O bottlenecks.
Optimize table/partition/chunk parallelism.
Tune workloads to meet migration-throughput objectives.
Identify bottlenecks across:
Hadoop Read → Spark → TaaS/Tokenization → Iceberg Write → Reconciliation
Coordinate technical remediation with infrastructure/storage/network teams where bottlenecks are outside the migration framework.
Apache Iceberg Engineering
Implement approved Iceberg table standards.
Develop/configure Iceberg tables and Spark→Iceberg processing.
Implement:
o partitioning,
o schema evolution,
o snapshots,
o file sizing,
o compaction,
o metadata maintenance,
o small-file remediation.
Troubleshoot Iceberg catalog/write/commit issues.
Optimize Iceberg table performance and storage characteristics.
Cyber-3 / Tokenization
Implement PNC-approved TaaS/Protegrity tokenization and retokenization patterns.
Integrate tokenization into Spark migration pipelines.
Implement batching, retry, timeout and recovery handling.
Troubleshoot tokenization failures/performance bottlenecks.
Measure impact of tokenization on end-to-end migration throughput.
Support Cyber/tokenization validation based on PNC-defined rules.
Migration Controls & Reconciliation
Lead implementation of PNC Data Movement Controls, including:
o checkpointing,
o idempotency,
o control totals,
o restart/recovery,
o retry,
o audit logging,
o exception handling.
Build reusable source-to-target reconciliation capabilities covering:
o row counts,
o schema validation,
o partition completeness,
o control totals,
o aggregates,
o hash/checksum where appropriate,
o null/duplicate validation,
o tokenization validation.
Isolate discrepancies at table/partition level.
Enable selective reprocessing rather than full-table reruns.
Produce technical evidence required for migration acceptance.
Mandatory Skills
Strong hands-on expertise in:
Python
PySpark
Apache Spark / Spark SQL
Hadoop / HDFS
Hive / HiveQL / Hive Metastore
SQL
Apache Iceberg
Data modelling
Source-to-target mapping
Parquet
Linux
Git
Large-volume ETL/data migration
Data validation/reconciliation
Spark performance tuning
Distributed-system troubleshooting.
Data Modelling
Strong experience with:
logical and physical data modelling,
source-to-target mapping,
schema/data-type mapping,
entity relationships,
keys and referential relationships,
dimensional/relational modelling concepts,
large-scale analytical data models.
Technical Leadership
Demonstrated experience:
leading Data Engineering teams,
technical estimation/planning,
detailed solution design,
code/design reviews,
mentoring engineers,
production troubleshooting,
root-cause analysis,
communicating technical issues with architects/customer teams.
Preferred Skills
Apache Polaris/Nessie/Glue.
OpenShift/Kubernetes/Spark Operator.
S3-compatible object storage.
NetApp StorageGRID/ONTAP.
TaaS/Protegrity or equivalent tokenization technology.
Trino.
PostgreSQL.
Control-M/Airflow.
CI/CD for Spark/Python applications.
Banking/financial-services experience.
PB-scale data migration.
Bronze/Silver/Gold modelling and implementation.
HiveQL→Spark SQL modernization.