Sign-In
Cart
Data Engineering with Databricks Cookbook: Build effective data and AI solutions using Apache Spark, Databricks, and Delta Lake
Paperback
ISBN13: 9781837633357
Publisher: Packt Publishing
Published: May 31 2024
Pages: 438
Weight: 1.65
Height: 0.89 Width: 7.50 Depth: 9.25
Language: English
Work through 70 recipes for implementing reliable data pipelines with Apache Spark, optimally store and process structured and unstructured data in Delta Lake, and use Databricks to orchestrate and govern your data
Key Features:
- Learn data ingestion, data transformation, and data management techniques using Apache Spark and Delta Lake
- Gain practical guidance on using Delta Lake tables and orchestrating data pipelines
- Implement reliable DataOps and DevOps practices, and enforce data governance policies on Databricks
- Purchase of the print or Kindle book includes a free PDF eBook
Book Description:
Written by a Senior Solutions Architect at Databricks, Data Engineering with Databricks Cookbook will show you how to effectively use Apache Spark, Delta Lake, and Databricks for data engineering, starting with comprehensive introduction to data ingestion and loading with Apache Spark.
What makes this book unique is its recipe-based approach, which will help you put your knowledge to use straight away and tackle common problems. You'll be introduced to various data manipulation and data transformation solutions that can be applied to data, find out how to manage and optimize Delta tables, and get to grips with ingesting and processing streaming data. The book will also show you how to improve the performance problems of Apache Spark apps and Delta Lake. Advanced recipes later in the book will teach you how to use Databricks to implement DataOps and DevOps practices, as well as how to orchestrate and schedule data pipelines using Databricks Workflows. You'll also go through the full process of setup and configuration of the Unity Catalog for data governance.
By the end of this book, you'll be well-versed in building reliable and scalable data pipelines using modern data engineering technologies.
What You Will Learn:
- Perform data loading, ingestion, and processing with Apache Spark
- Discover data transformation techniques and custom user-defined functions (UDFs) in Apache Spark
- Manage and optimize Delta tables with Apache Spark and Delta Lake APIs
- Use Spark Structured Streaming for real-time data processing
- Optimize Apache Spark application and Delta table query performance
- Implement DataOps and DevOps practices on Databricks
- Orchestrate data pipelines with Delta Live Tables and Databricks Workflows
- Implement data governance policies with Unity Catalog
Who this book is for:
This book is for data engineers, data scientists, and data practitioners who want to learn how to build efficient and scalable data pipelines using Apache Spark, Delta Lake, and Databricks. To get the most out of this book, you should have basic knowledge of data architecture, SQL, and Python programming.
Table of Contents
- Data Ingestion and Data Extraction with Apache Spark
- Data Transformation and Data Manipulation with Apache Spark
- Data Management with Delta Lake
- Ingesting Streaming Data
- Processing Streaming Data
- Performance Tuning with Apache Spark
- Performance Tuning in Delta Lake
- Orchestration and Scheduling Data Pipeline with Databricks Workflows
- Building Data Pipelines with Delta Live Tables
- Data Governance with Unity Catalog
- Implementing DataOps and DevOps on Databricks
Also in
General Computers
This Program Is Brought to You by . . .: Distributing Television News Online
Braun, Joshua A.
Paperback
The Year in Tech, 2025: The Insights You Need from Harvard Business Review
Review, Harvard Business
Webb, Amy
Farri, Elisa
Paperback
The Technological Republic: Hard Power, Soft Belief, and the Future of the West
Zamiska, Nicholas W.
Karp, Alexander C.
Hardcover
AI Snake Oil: What Artificial Intelligence Can Do, What It Can't, and How to Tell the Difference
Narayanan, Arvind
Kapoor, Sayash
Hardcover
Hbr's 10 Must Reads on AI (with Bonus Article How to Win with Machine Learning by Ajay Agrawal, Joshua Gans, and AVI Goldfarb)
Review, Harvard Business
Davenport, Thomas H.
Iansiti, Marco
Paperback
Mindmasters: The Data-Driven Science of Predicting and Changing Human Behavior
Matz, Sandra
Hardcover
Generative Ai: The Insights You Need from Harvard Business Review
Mollick, Ethan
Cremer, David De
Review, Harvard Business
Paperback
The Coming Wave: Technology, Power, and the Twenty-First Century's Greatest Dilemma
Suleyman, Mustafa
Hardcover
AI for Educators: Learning Strategies, Teacher Efficiencies, and a Vision for an Artificial Intelligence Future
Miller, Matt
Paperback
More Human: How the Power of AI Can Transform the Way You Lead
Carter, Jacqueline
Hougaard, Rasmus
Hardcover
The AI Con: How to Fight Big Tech's Hype and Create the Future We Want
Bender, Emily M.
Hanna, Alex
Hardcover
Teaching with AI: A Practical Guide to a New Era of Human Learning
Watson, C. Edward
Bowen, José Antonio
Paperback
Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
Kleppmann, Martin
Paperback
Brave New Words: How AI Will Revolutionize Education (and Why That's a Good Thing)
Khan, Salman
Hardcover
Python Crash Course, 3rd Edition: A Hands-On, Project-Based Introduction to Programming
Matthes, Eric
Paperback
AI Valley: Microsoft, Google, and the Trillion-Dollar Race to Cash in on Artificial Intelligence
Rivlin, Gary
Hardcover
Princeton Review AP Computer Science a Prep, 8th Edition: 5 Practice Tests + Complete Content Review + Strategies & Techniques
The Princeton Review
Paperback
Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence
Crawford, Kate
Paperback
Banking on (Artificial) Intelligence: Navigating the Realities of AI in Financial Services
Lau, Theodora
Paperback
Digital Dharma: How AI Can Elevate Spiritual Intelligence and Personal Well-Being
Chopra, Deepak
Hardcover
The Black Swan: Second Edition: The Impact of the Highly Improbable: With a New Section: On Robustness and Fragility
Taleb, Nassim Nicholas
Paperback
Nexus: A Brief History of Information Networks from the Stone Age to AI (Large Print Edition)
Harari, Yuval Noah
Paperback
RHCSA Red Hat Enterprise Linux 9: Training and Exam Preparation Guide (EX200), Third Edition
Ghori, Asghar
Paperback
AI for Life: 100+ Ways to Use Artificial Intelligence to Make Your Life Easier, More Productive...and More Fun!
Quillian, Celia
Paperback
The Death of Expertise: The Campaign Against Established Knowledge and Why It Matters
Nichols, Tom
Paperback
Minecraft: Exploded Builds: Medieval Fortress: An Official Mojang Book
Mojang Ab
The Official Minecraft Team
Paperback
Fans First: Change The Game, Break the Rules & Create an Unforgettable Experience
Cole, Jesse
Paperback
Hands-On Large Language Models: Language Understanding and Generation
Grootendorst, Maarten
Alammar, Jay
Paperback
Minecraft: Guide Collection 4-Book Boxed Set (Updated): Survival (Updated), Creative (Updated), Redstone (Updated), Combat
Mojang Ab
The Official Minecraft Team
Hardcover
The Thinking Machine: Jensen Huang, Nvidia, and the World's Most Coveted Microchip
Witt, Stephen
Hardcover
Embedded Systems with ARM Cortex-M Microcontrollers in Assembly Language and C: Fourth Edition
Zhu, Yifeng
Paperback
The Chaos Machine: The Inside Story of How Social Media Rewired Our Minds and Our World
Fisher, Max
Paperback
What Tech Calls Thinking: An Inquiry Into the Intellectual Bedrock of Silicon Valley
Daub, Adrian
Paperback
R for Data Science: Import, Tidy, Transform, Visualize, and Model Data
Wickham, Hadley
Grolemund, Garrett
Cetinkaya-Rundel, Mine
Paperback
Recoding America: Why Government Is Failing in the Digital Age and How We Can Do Better
Pahlka, Jennifer
Hardcover
Laptops for Seniors in Easy Steps, 9th Edition: Updated to Cover All Laptops with the Windows 11 2024 Update
Vandome, Nick
Paperback
Computer Science: An Illustrated History of the World's Smartest Machines (100 Ponderables)
Jackson, Tom
Hardcover
Practical Charts: The Essential Guide to Creating Clear, Compelling Charts for Reports and Presentations
Desbarats, Nicholas P.
Paperback
The Mechanic and the Luddite: A Ruthless Criticism of Technology and Capitalism
Sadowski, Jathan
Paperback
Exploring Windows 11 - 2024 Edition: The Illustrated, Practical Guide to Using Microsoft Windows
Wilson, Kevin
Paperback
Fundamentals of Data Engineering: Plan and Build Robust Data Systems
Housley, Matt
Reis, Joe
Paperback
Prompt Engineering for Generative AI: Future-Proof Inputs for Reliable AI Outputs
Phoenix, James
Taylor, Mike
Paperback
Algorithms to Live by: The Computer Science of Human Decisions
Christian, Brian
Griffiths, Tom
Paperback
Arduino: 101 Beginners Guide: How to get started with Your Arduino (Tips, Tricks, Projects and More!)
Savasgard, Erik
Paperback
Isc2 Cissp Certified Information Systems Security Professional Official Study Guide & Practice Tests Bundle
Gibson, Darril
Chapple, Mike
Stewart, James Michael
Paperback
You Look Like a Thing and I Love You: How Artificial Intelligence Works and Why It's Making the World a Weirder Place
Shane, Janelle
Paperback
Human + Machine, Updated and Expanded: Reimagining Work in the Age of AI
Daugherty, Paul R.
Wilson, H. James
Hardcover
Verified: How to Think Straight, Get Duped Less, and Make Better Decisions about What to Believe Online
Wineburg, Sam
Caulfield, Mike
Paperback
Exploring Apple Mac - Sequoia Edition: The Illustrated, Practical Guide to Using MacOS
Wilson, Kevin
Paperback
Crypto Confidential: Winning and Losing Millions in the New Frontier of Finance
Eliason, Nathaniel
Hardcover
Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications
Huyen, Chip
Paperback
Building AI-Powered Products: The Essential Guide to AI and Genai Product Management
Nika, Marily
Paperback
Your Stone Age Brain in the Screen Age: Coping with Digital Distraction and Sensory Overload
Cytowic, Richard E.
Hardcover
How to Teach AI: Weaving Strategies and Activities Into Any Content Area
Poth, Rachelle Dené
Paperback
Future Ready: The Four Pathways to Capturing Digital Value
Sebastian, Ina M.
Woerner, Stephanie L.
Weill, Peter
Hardcover
Automate the Boring Stuff with Python, 2nd Edition: Practical Programming for Total Beginners
Sweigart, Al
Paperback
AP Computer Science a Premium, 12th Edition: Prep Book with 6 Practice Tests + Comprehensive Review + Online Practice
Teukolsky, Roselyn
Barron's Educational Series
Paperback
Irresistible: The Rise of Addictive Technology and the Business of Keeping Us Hooked
Alter, Adam
Paperback
The Experimentation Machine: Finding Product-Market Fit in the Age of AI
Bussgang, Jeffrey J.
Hardcover
Ocp Oracle Certified Professional Java Se 21 Developer Study Guide
Selikoff, Scott
Boyarsky, Jeanne
Paperback
The Algorithm: How AI Decides Who Gets Hired, Monitored, Promoted, and Fired and Why We Need to Fight Back Now
Schellmann, Hilke
Hardcover
Better Data Visualizations: A Guide for Scholars, Researchers, and Wonks
Schwabish, Jonathan
Paperback
ITIL(R) 4 Essentials: Your essential guide for the ITIL 4 Foundation exam and beyond
Agutter, Claire
Paperback