Data & AI Engineer

Sobre a oportunidade

This position is listed on behalf of a partner company, which manages all applications and next steps.

Local

São Paulo - SP

Remoto

Responsabilidades

Design, develop, and optimize scalable ETL/ELT pipelines using Databricks, Apache Spark, and Delta Lake across cloud environments, including AWS, Azure, and Google Cloud Platform.

Build robust workflows for processing structured and unstructured data, automating ingestion, storage, and feature engineering to support analytics and real-time AI workloads.

Tune Spark jobs and data processing workflows to improve latency, throughput, scalability, and cloud cost efficiency across client environments.

Collaborate with machine learning engineers and data scientists to operationalize AI models within Databricks, using MLflow for experiment tracking, model versioning, and deployment.

Implement automated deployment pipelines and Infrastructure-as-Code practices using tools such as Terraform and Databricks CLI to ensure reliable, repeatable production releases.

Establish and maintain data access controls, security policies, and governance standards using Unity Catalog and cloud-native capabilities, supporting requirements such as GDPR, HIPAA, and SOC 2.

Work directly with client technology leaders, product managers, architects, and engineering teams to translate complex data and AI requirements into practical technical solutions and measurable business outcomes.

Produce clear technical documentation and coordinate smooth solution handoffs to client engineering teams, ensuring maintainability and long-term operational success.

Requisitos

At least three years of experience in cloud-native data engineering, including a minimum of two years of hands-on experience with Databricks, Apache Spark, Delta Lake, and MLflow.

Strong proficiency in Python, SQL, and Spark-based frameworks, with experience designing and maintaining scalable data processing solutions.

Practical experience building data architectures across cloud environments and implementing modern Lakehouse patterns for enterprise data workloads.

Familiarity with MLflow and the processes required to track, version, deploy, and operationalize machine learning models.

Experience implementing CI/CD pipelines and Infrastructure-as-Code solutions using Terraform, Databricks CLI, or comparable tools.

An active Databricks Certified Data Engineer Associate or Professional certification, or Databricks Machine Learning Professional certification, is strongly preferred.

Proven experience working in fast-paced Agile or DevOps consulting environments, delivering cloud-agnostic solutions and adapting to evolving client requirements.

Strong communication skills, with the ability to explain technical concepts clearly, work effectively with multidisciplinary teams, and maintain productive client relationships.

A proactive, accountable approach to problem-solving, with a commitment to engineering quality, continuous learning, and delivering measurable outcomes.

Benefícios

Remote-First Work: The flexibility to work remotely while collaborating with a globally connected team.

Collaborative Environment: The opportunity to work alongside skilled professionals in a supportive, innovative, and fast-paced setting.

Technical Growth: Exposure to modern Lakehouse architectures, cloud platforms, data engineering, MLOps, and production AI solutions.

Meaningful Client Impact: The opportunity to build scalable, production-ready solutions that address real business challenges.

Inclusive Workplace: A culture that values diversity, equal opportunity, collaboration, curiosity, and individual initiative.