Job Description Job Description Senior Data Engineer - AI, Automation & Databricks
Location: Houston, TX
Work Arrangement: Houston-based / Onsite as required
Experience: 8+ years preferred
Employment Type: Contract / Full-Time, based on project requirement
Job Summary
We are looking for a highly experienced and innovative Senior Data Engineer to assess, modernize, and optimize enterprise Data Engineering processes while driving the adoption of automation, Artificial Intelligence, and Agentic AI.
The ideal candidate will combine strong hands-on Data Engineering expertise with a proactive problem-solving mindset. This individual will evaluate existing engineering processes, identify opportunities for AI and automation, develop Proofs of Concept (POCs), and help transition successful ideas into production-ready solutions.
The role requires deep experience with Databricks, AWS Redshift, modern Data Engineering architectures, AI-enabled engineering practices, and AI solutions across the Software Development Life Cycle (SDLC). The successful candidate should be comfortable working independently, challenging existing approaches, proposing innovative solutions, and collaborating with both technical and business stakeholders.
Key Responsibilities
- Assess existing Data Engineering platforms, workflows, pipelines, architecture, and operational processes and identify opportunities for optimization.
- Design and implement improvements to enhance data pipeline performance, scalability, reliability, maintainability, and operational efficiency.
- Identify opportunities to introduce automation and AI capabilities across Data Engineering processes.
- Develop Proofs of Concept (POCs) for AI- and automation-driven ideas and evaluate their technical and business feasibility.
- Drive successful POCs through implementation and production adoption in collaboration with engineering, architecture, security, and business teams.
- Design and implement solutions leveraging Agentic AI, Generative AI, and AI-assisted Data Engineering.
- Identify opportunities to apply AI throughout the Software Development Life Cycle, including:
- Requirements analysis
- Solution design
- Code generation and optimization
- Data pipeline development
- Automated testing
- Data quality validation
- Documentation
- Monitoring and observability
- Incident analysis and troubleshooting
- Build and optimize scalable Data Engineering solutions using Databricks.
- Design, develop, and optimize enterprise data workloads using AWS Redshift.
- Develop and optimize ETL/ELT pipelines supporting large-scale structured and unstructured data environments.
- Evaluate existing engineering practices and proactively recommend improvements rather than relying solely on predefined requirements.
- Identify repetitive or manual engineering activities that can be automated through scripts, workflows, AI agents, or orchestration platforms.
- Partner with Data Engineers, Data Architects, Data Scientists, AI/ML teams, application teams, and business stakeholders.
- Translate complex technical opportunities into clear business value and build stakeholder alignment around proposed solutions.
- Establish reusable engineering patterns, frameworks, accelerators, and best practices.
- Conduct technical evaluations of emerging AI and Data Engineering technologies and recommend appropriate adoption strategies.
- Support troubleshooting and root-cause analysis of complex Data Engineering and platform-related issues.
- Document technical designs, architecture decisions, POCs, implementation approaches, and operational procedures.
Required Technical Skills
- Strong hands-on experience in Data Engineering within large-scale enterprise environments.
- Advanced experience with Databricks.
- Strong experience with AWS Redshift.
- Strong understanding of:
- Data lakes and lakehouse architectures
- Data warehouses
- ETL/ELT architecture
- Distributed data processing
- Data pipeline orchestration
- Data quality and governance
- Performance optimization
- Hands-on experience with Apache Spark / PySpark.
- Strong programming experience with Python.
- Strong SQL development and query optimization skills.
- Experience working with AWS Data and Analytics services.
- Experience designing and supporting scalable cloud-based data platforms.
- Understanding of CI/CD and DevOps principles within Data Engineering environments.
AI & Automation Experience
Candidates should have practical experience applying AI beyond traditional analytics or machine learning use cases.
Preferred experience includes:
- Agentic AI / AI Agents
- Generative AI applications within Data Engineering
- LLM-based engineering solutions
- AI-assisted code generation and review
- Automated pipeline troubleshooting
- Intelligent Data Quality validation
- AI-assisted data transformation and mapping
- AI-driven monitoring and anomaly detection
- Automated documentation and metadata generation
- AI-enabled SDLC processes
- Prompt engineering and LLM integration
- Development of AI/automation POCs
- Integration of AI solutions into production engineering workflows
Experience with AI development frameworks such as LangChain, LangGraph, Semantic Kernel, Amazon Bedrock, OpenAI/Azure OpenAI, or similar technologies would be advantageous.
Required Professional Skills
- Strong analytical and problem-solving capabilities.
- Demonstrated ability to independently identify technical and process improvement opportunities.
- Strong innovation mindset with a track record of proposing and implementing new ideas.
- Ability to operate effectively in ambiguous environments without requiring detailed direction.
- Strong communication and stakeholder-management skills.
- Ability to collaborate across technical teams, business stakeholders, leadership, and architecture groups.
- Ability to explain complex technical concepts to non-technical stakeholders.
- Strong ownership mentality with the ability to take ideas from concept POC validation implementation.
- Ability to work effectively as part of a collaborative engineering team.
Preferred Qualifications
- 8+ years of overall Data Engineering, Data Platform, or related engineering experience.
- Significant hands-on experience with enterprise Data Engineering architectures.
- Experience leading Data Engineering modernization or optimization initiatives.
- Demonstrated experience building AI- or automation-enabled engineering solutions.
- Experience working within large, complex enterprise environments.
- Experience influencing architecture or engineering standards across multiple teams.
- Exposure to airline, transportation, travel, or other large-scale transactional environments would be advantageous.
Ideal Candidate Profile
The ideal candidate is not simply someone who can build pipelines from predefined requirements. We are looking for an engineer who can evaluate the existing environment, identify what can be improved, and proactively develop solutions.
This person should be able to:
- Understand the current Data Engineering landscape quickly.
- Identify inefficiencies and improvement opportunities.
- Determine where AI or automation can create measurable value.
- Build working POCs rather than limiting recommendations to conceptual presentations.
- Demonstrate the value of proposed solutions to technical and business stakeholders.
- Convert successful concepts into scalable production implementations.
- Challenge existing engineering practices constructively.
- Work collaboratively and fit effectively within a multidisciplinary team.
Core Skills
Databricks AWS Redshift Data Engineering Python PySpark SQL AWS ETL/ELT Data Lakehouse Agentic AI Generative AI LLMs AI Automation AI across SDLC POC Development Data Platform Optimization Stakeholder Management Innovation
Company Description We provide Management Consulting, Information Technology Consulting, and Sourcing Advisory Services.
Our management consulting services focus on our client's most critical issues and opportunities: strategy, marketing, organization, operations, technology, and mergers & acquisitions, across all industries and geographies.
We bring deep, functional expertise, but are known for our holistic perspective: we capture value across boundaries and between the silos of any organization. We have proven a multiplier effect by optimizing the sum of the parts, not just the individual pieces.
In today's dynamic business environment there is a strong need for the organizations to become globally competitive. Enterprise Resource Planning (ERP) has become an important software tool for managing business applications across various industries . click apply for full job details