CoreHRx Jobs

Apply in one click, save jobs and take free mock tests.

Register free Log in
Hiring? Post jobs with CoreHRx
My account
user

View & update profile

Senior Platform Engineer

Insight Global

Full time Remote
Work mode
Remote
Posted 3 days ago Last checked 1 day ago

Free · takes a minute · your resume fills your profile

CoreHRx never asks candidates for money. If anyone does, .

Role at a glance

You will
  • Design, develop and maintain scalable batch and streaming pipelines with Azure Data Factory, Azure Databricks and Azure Function Apps,
  • integrating data from databases, ERP and CRM platforms, files, APIs and event streams.
  • Build PySpark and Spark SQL transformations that turn raw data into clean, conformed and trusted datasets, extending the team's
Education
Bachelor's degree
Must-have skills
Apache SparkAzure DevOpsCI/CDData EngineeringData ModelingDatabricks
Work mode
Remote

How well do your skills match this job?

Sign in to see your fit: the skills, experience and location this job asks for, checked against your profile.

Job description

Location: Remote, India

Job Description
Our client is is looking for experienced Data Engineers to join our data engineering team. You will design, build and operate the pipelines that power our enterprise data lake on Microsoft Azure, delivering reliable, high-quality data for operational reporting, analytics and machine learning across our parcel network. Working primarily with Azure Data Factory and Azure Databricks, you will own pipelines from source systems through to curated,
business-ready datasets.
The ideal candidate combines strong hands-on engineering skills with a focus on performance, cost efficiency and
data quality, and works effectively with analytics, BI, data science and platform teams.
Key Responsibilities:

  • Design, develop and maintain scalable batch and streaming pipelines with Azure Data Factory, Azure Databricks and Azure Function Apps,

integrating data from databases, ERP and CRM platforms, files, APIs and event streams.

  • Build PySpark and Spark SQL transformations that turn raw data into clean, conformed and trusted datasets, extending the team's

configuration-driven ingestion framework.

  • Design Delta Lake tables and lakehouse layers (raw, curated and reporting), with partitioning, clustering and data models aligned to how

the data is consumed.

  • Optimize the performance and cost of Spark workloads by tuning joins, file layout and cluster configuration, and eliminate unnecessary

reprocessing.

  • Implement monitoring and alerting for pipeline failures, run duration and data freshness, and lead root-cause analysis and resolution of

production incidents.

  • Implement data quality checks, schema evolution handling and Unity Catalog access controls to keep data accurate, secure and well

governed.

  • Prepare feature tables and training datasets, and support machine learning workflows on Databricks using MLflow and Unity Catalog.
  • Deliver curated datasets to analysts, data scientists and Power BI developers, and help optimize semantic models and report refresh

performance.

  • Package and deploy jobs and pipelines with Databricks Asset Bundles through Azure DevOps CI/CD, following the team's code review,

testing and documentation standards.

  • Mentor junior engineers and contribute to the team's engineering standards

Skills and Requirements

  • 8-12 years in data engineering, with hands-on Azure Databricks and Azure Data Factory experience in production
  • Advanced Python, PySpark and SQL, and strong Delta Lake knowledge
  • Proven experience tuning Spark workloads for performance and cost
  • Experience designing incremental and streaming pipelines and data models
  • Hands-on Databricks machine learning experience: MLflow, feature engineering and model training workflows
  • Experience with Databricks Asset Bundles and CI/CD in Azure DevOps
  • Experience building Azure Function Apps for API and event-driven ingestion
  • Production monitoring, alerting and incident management
  • Bachelor's degree in a technical field, or equivalent experience
  • Spark Structured Streaming, Kafka or Azure Event Hubs
  • Lakeflow pipelines and jobs, and Databricks serverless compute
  • Unity Catalog administration and Databricks system tables
  • Power BI (data modeling, DAX) and Microsoft Fabric
  • Terraform or another infrastructure as code tool
  • Databricks Data Engineer or Machine Learning certification

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to HR@insightglobal.com.

Job data: Headcount Labs (headcountlabs.com), ODbL 1.0. The job description belongs to the employer. Location data © OpenStreetMap contributors and Overture Maps.

Key skills

Required

Apache SparkAzure DevOpsCI/CDData EngineeringData ModelingDatabricksFeature EngineeringIncident ManagementKafkaMachine LearningMicrosoft AzurePower BIPythonSQLTerraform

Job market insights

501active Apache Spark jobs in India
841active Azure DevOps jobs in India
3,821active CI/CD jobs in India
47 more open jobs at Insight Global

Jobs asking for Apache Spark in India also ask for

  • Python81%
  • SQL71%
  • Databricks52%
  • Data Engineering51%
  • AWS42%

From live job posts on CoreHRx, updated 20 Nov 2026. Skill figures count posts that list their skills; salary figures only posts that show a salary.

Questions about this job

Where is this job based?

It is a remote job.

What kind of job is this?

It is a full time job.

This guide is for freshers in India preparing for their first data science or machine learning interview, and for analysts and engineers with a few years of experience moving into ML roles. It has 60 questions with model answers in three levels: the basics every fresher is asked, then the practical questions on metrics, validation, leakage and tuning that come up for 1-4 years of experience, and finally senior questions on model internals, deployment, drift and explaining models to business teams. The worked numbers in the answers are small enough to check by hand. Try each answer aloud before reading ours, then take the mock test to practise under time pressure.

Questions you may be asked

  1. What is machine learning, and how is it different from traditional programming?
  2. What is the difference between supervised and unsupervised learning?
  3. How do you decide whether a problem is classification or regression?
  4. Why do we split data into training, validation and test sets?

Get IT jobs you can do remotely by email

Create a free profile and we email you new IT jobs you can do remotely, at most once a week.

Similar jobs

Free · takes a minute · your resume fills your profile