EP 50: Evolution of Enterprise Data Engineering in Gen AI Era

Shashank Akinapalli is a Senior Data Engineer at TCS where he leads cloud data lakehouse initiatives and real-time streaming architectures for supply chain intelligence. With 14 years of experience across IBM DataStage, Databricks, Snowflake, dbt, AWS, and Spark-based ecosystems, he has previously worked with USAA in banking and Blue Cross Blue Shield in healthcare. IN THIS EPISODE:▪ The biggest architectural shift when migrating from IBM DataStage to a modern cloud lakehouse — and why the technical migration is the easier half, while reconstructing encoded business knowledge is the real challenge▪ How to choose between Snowflake and Databricks: Databricks for heavy transformation and Python workloads, Snowflake for structured SQL and BI, and where dbt fits as the layer that eliminates repetitive SQL code▪ The real UNFI example: how batch processing caused inventory misrepresentation (800 units sold, system still showing 1,000 available), and how real-time Databricks pipelines eliminated that problem and unlocked fraud detection on top▪ Data governance across three regulated domains: the four pillars — data quality, redundancy, least-privilege access, and lineage — and the three questions every framework must answer first▪ Why CI/CD transformed data engineering: from manual, multi-team deployments with risky rollbacks to automated quality checks, Git version control, and the specific silent failure that finally convinces a team to invest▪ How the data engineer role shifts over the next three years: from pipeline development to data product engineering, from reactive operations to AI-predicted failure prevention, and from human dashboards to AI agents making autonomous predictions▪ Practical advice for data engineers who want to stay relevant: Python, PySpark, cloud architecture, and understanding the full data lifecycle from ingestion to consumption FIND SHASHANK:LinkedIn: https://www.linkedin.com/in/shashank-a-2aa629119/Twitter / Instagram: @shashi001gitHappy to answer questions on enterprise data warehouse, data engineering, and cloud architecture modernisation — reach out directly via LinkedIn or email.DATASCIENCEWITHSAM:If you work in data engineering and are navigating a legacy modernisation right now — this is the episode to send your team. Subscribe on Apple Podcasts, Spotify, Amazon Music, iHeartRadio, Podbean, and YouTube. If you enjoyed this episode, please share it with your network. DataScienceWithSam is always looking for new guests for captivating discussions — if you have a topic you'd like to discuss on a 30–45 minute podcast, feel free to reach out.