Company Profile:
Founded in 1976, CGI is among the largest independent IT and business consulting services firms in the world. With 94,000 consultants and professionals across the globe, CGI delivers an end-to-end portfolio of capabilities, from strategic IT and business consulting to systems integration, managed IT and business process services and intellectual property solutions. CGI works with clients through a local relationship model complemented by a global delivery network that helps clients digitally transform their organizations and accelerate results. CGI Fiscal 2024 reported revenue is CA$14.68 billion and CGI shares are listed on the TSX (GIB.A) and the NYSE (GIB). Learn more at cgi.com. #LI-SB15
Position: Data Lake Platform Developer
Experience: 5 to 10 Years
Category: Business Intelligence
Shift: General
Location: Mum / Chn / Hyd
Employment Type: Permanent
Education Qualification: Bachalors
Position Description:
We are looking for a highly experienced, hands-on Data Lake Platform Developer to help build, enhance, and maintain our Data Lake platform. In this role, you will be heavily involved in coding, researching technical solutions, implementing frameworks, and working with various third-party and homegrown tools to deliver end-to-end data solutions.
Your primary focus will be developing and maintaining complex data pipelines that stream real-time data from ERP systems and databases into Snowflake. You will work on a diverse technical stack, handling everything from real-time Change Data Capture (CDC) and fallback synchronization to DevOps automation, monitoring, and supporting the integration of downstream reporting tools like Power BI.
Your future duties and responsibilities
Platform Development & Tool Integration
Solution Implementation: Write code to build, enhance, and maintain the Data Lake platform. Research and implement homegrown frameworks and integrate third-party tools to meet platform requirements.
Back-End Development: Develop and maintain server-side logic, APIs, and backend integrations interacting with databases like Oracle, SQL Server, and PostgreSQL.
Data Engineering & Real-Time Streaming
Pipeline Development: Implement and maintain moderate to highly complex end-to-end data pipelines utilizing Python, Kafka, and Debezium.
Change Data Capture (CDC): Work on real-time CDC streaming pipelines from ERP systems into the Snowflake data lake, utilizing tools and concepts like Oracle LogMiner, XStream, etc.
Snowflake Development: Write, optimize, and maintain advanced SQL, Stored Procedures, and complex scripting within Snowflake.
Platform Resilience, DevOps & Monitoring
Reliability & Fallbacks: Develop and maintain scripts for initial data loads, as well as alternate data synchronization processes to serve as fallbacks for pipeline interruptions.
DevOps & Automation: Contribute to DevOps initiatives by automating the deployment and CI/CD processes for data pipelines.
Monitoring: Implement and configure end-to-end monitoring solutions to track pipeline health and data flow.
BI Integration & Security
Reporting Artifact Frameworks: Develop and support tools/frameworks that aid in the deployment and development of reporting artifacts (such as Power BI).
BI Embedding: Write the necessary backend code and integrations to help embed BI and reporting artifacts directly into our ERP applications.
Security Implementation: Implement data security policies, row-level security, and access controls for downstream consumption based on established guidelines.
Code Quality & Agile Practices
Code Quality: Write clean, well-documented, maintainable, and efficient Python/SQL code based on detailed designs and best practices.
Performance Optimization: Identify and resolve performance bottlenecks in both data ingestion scripts and database queries to ensure smooth processing.
Testing and Debugging: Conduct unit tests, integration tests, and debugging to ensure the reliability and stability of the data pipelines and related applications.
Version Control: Utilize version control systems (e.g., Git) to manage and collaborate on code repositories.
Collaboration & Agile: Participate in Agile development processes, including sprint planning, daily stand-ups, and retrospectives, working closely with cross-functional teams.
Continuous Learning: Stay updated on the latest data engineering trends, streaming tools, and technologies, and apply them to your daily development tasks.
Required qualifications to be successful in this role
Good to have skills:
Solid hands-on experience as a Senior Developer/Data Engineer working on complex data platforms.
Strong programming proficiency in Python and Advanced SQL.
Proven experience developing in Snowflake (writing stored procedures, scripting, and optimizing queries).
Practical experience building and maintaining real-time data streaming and CDC pipelines using Kafka and Debezium (familiarity with Oracle LogMiner/XStream is a strong plus).
Experience working with relational databases (Oracle, PostgreSQL, SQL Server).
Hands-on experience with DevOps automation, CI/CD pipelines, and application monitoring tools.
Strong independent research skills with a track record of successfully implementing new frameworks or third-party tools into an existing codebase
Must to have skill:
Knowledge of .NET (C# / .NET Core) is a plus.
Familiarity with the backend integration of Power BI (or similar tools) and embedding analytics into existing applications.
#LI-GN3
CGI is an equal opportunity employer. In addition, CGI is committed to providing accommodation for people with disabilities in accordance with provincial legislation. Please let us know if you require reasonable accommodation due to a disability during any aspect of the recruitment process and we will work with you to address your needs.
Together, as owners, let’s turn meaningful insights into action.
Life at CGI is rooted in ownership, teamwork, respect and belonging. Here, you’ll reach your full potential because…
You are invited to be an owner from day 1 as we work together to bring our Dream to life. That’s why we call ourselves CGI Partners rather than employees. We benefit from our collective success and actively shape our company’s strategy and direction.
Your work creates value. You’ll develop innovative solutions and build relationships with teammates and clients while accessing global capabilities to scale your ideas, embrace new opportunities, and benefit from expansive industry and technology expertise.
You’ll shape your career by joining a company built to grow and last. You’ll be supported by leaders who care about your health and well-being and provide you with opportunities to deepen your skills and broaden your horizons.
Come join our team—one of the largest IT and business consulting services firms in the world.