Job Description
Job Responsibilities:
Experience: 8 - 12 years
Location: Bangalore, Karnataka, India
Description
The candidate for this data engineer position brings with them 8 to 12 years of extensive experience in the field, making them a highly skilled and knowledgeable professional. They have a strong expertise in Azure Data Engineering, having worked with the platform for several years and gaining a deep understanding of its capabilities and functionalities. Their proficiency in Azure Data Engineering is complemented by their additional skills in ADF (Azure Data Factory) and Databricks, making them a well-rounded data engineer with a diverse set of technical skills. They have a proven track record of designing, building, and maintaining complex data pipelines and systems using these technologies, and have a thorough understanding of data warehousing, data modeling, and ETL processes. With their extensive experience and expertise, the candidate is well-equipped to handle any data engineering challenges and deliver high-quality solutions to drive business success.
Responsibilities
- Develop and maintain data pipelines and ETL processes using Azure Data Factory (ADF) and Databricks
- Design and implement data storage solutions on Azure, including Azure SQL Database, Azure Data Lake Storage, and Azure Blob Storage
- Create and manage data models and data warehouses using Azure Synapse Analytics
- Collaborate with cross-functional teams to gather and analyze business requirements and translate them into technical solutions
- Troubleshoot and optimize data processes to ensure high performance and data quality
- Implement data security measures to protect sensitive information
- Stay updated with the latest data engineering technologies and best practices
- Lead and mentor junior data engineers in the team
- Provide technical guidance and support to other teams and stakeholders
- Participate in code reviews and ensure adherence to coding standards and best practices
- Collaborate with DevOps teams to deploy and manage data pipelines and processes
- Conduct data analysis and provide insights to support business decisions
- Develop and maintain documentation for data processes and solutions
- Identify and troubleshoot data-related issues and provide timely resolutions
Skills
- 8 to 12 years of experience as a data engineer
- Strong expertise in Azure Data Engineering
- Proficient in designing, developing, and maintaining data pipelines in Azure
- Expertise in using Azure Data Factory (ADF) for data integration and orchestration
- Experience with Azure Databricks for data processing and analytics
- In-depth knowledge of SQL and NoSQL databases
- Experience with data warehousing and data modeling
- Familiarity with ETL/ELT processes and tools
- Ability to optimize and troubleshoot data pipelines for performance and scalability
- Experience with version control and deployment tools such as Git and Azure DevOps
- Strong understanding of cloud computing concepts and architecture
- Experience with data security and privacy best practices
- Excellent problem-solving and analytical skills
- Strong communication and collaboration skills
- Experience working in an Agile environment
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field
