The position is for our client in Dubai.
The Azure data engineer focuses on data-related tasks in Azure.
Responsible for design, development, and architecture of Microsoft Azure Cloud solutions.
Experience with architecting and designing technical solutions using combination of Azure IaaS and PaaS
• Experience with Cloud Platforms such as Azure, AWS , Google Cloud etc.
• Experience on Migrating SQL database to Azure data Lake, Azure data lake Analytics, Azure SQL Database, Azure Synapse, Data Bricks and Azure SQL Data warehouse.
• Experience in Deploying Distributed Messaging Systems using KAFKA, or other messaging System.
• Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation and aggregation from multiple file formats for analysing& transforming the data to uncover insights into the customer usage patterns.
• Good understanding of Spark Architecture including Spark Core, Spark SQL, Data Frames, Spark Streaming, Driver Node, Worker Node, Stages, Executors and Tasks.
• Good understanding of Big Data Hadoop and Yarn architecture along with various Hadoop Demons such as Job Tracker, Task Tracker, Name Node, Data Node, Resource/Cluster Manager, and Kafka (distributed stream-processing) .
