DATA-ENG-ASSOC Sample Questions & Answers

The primary purpose of the Silver layer is to store cleansed, validated, and enriched data. This code snippet moves data directly from the source (Bronze) to Silver using a blind overwrite without any intermediate transformation steps. This violates the core principle of the Medallion architecture, as it bypasses the crucial data quality and shaping processes that should occur between the Bronze and Silver layers.

To create a physical table that can be queried later, @dlt.table is the correct decorator. To enforce a quality rule where invalid records are dropped while allowing the pipeline to continue, @dlt.expect_or_drop is the appropriate function. @dlt.expect_or_fail would halt the pipeline, and @dlt.expect would only record the violation statistics without dropping or failing.

Delta Sharing is the open protocol specifically designed for sharing live data from Databricks to any external consumer, regardless of their platform. It works by creating shares and recipients, then providing the recipient with a secure, one-time activation link to download a credential file. This allows them to access the shared data using various connectors (like Power BI, Tableau, pandas) without needing a Databricks account. Lakehouse Federation is for querying external data sources from within Databricks, not sharing data out.

Serverless SQL warehouses are ideal for this use case. They provide instant compute, automatically scale up and down to handle concurrent query loads, and eliminate the need for administrators to manage cluster configurations. This directly addresses the requirements for performance on variable workloads and reduced management overhead. A job cluster is for automated jobs, and an all-purpose cluster requires manual management and is less efficient for handling concurrent SQL queries.

The merge operation is specifically designed for this type of 'upsert' (update or insert) scenario. It allows you to define conditions for matching records between a source DataFrame and a target Delta table. The whenMatchedUpdate clause specifies the update logic for existing records, while the whenNotMatchedInsert clause handles the insertion of new records, all within a single, atomic transaction.

The 'Repair and Rerun' feature is designed for this exact purpose. It allows you to resume a failed workflow run from the point of failure. It intelligently skips the tasks that have already completed successfully (like ingest_data) and reruns only the failed tasks and their downstream dependencies. This is the most efficient method as it avoids re-processing data unnecessarily.

The Databricks Repos UI provides core Git functionality like commit, pull, push, and branching. However, for complex operations like resolving merge conflicts that Git cannot handle automatically, the user must pull the changes to their local machine, use standard Git tools to resolve the conflicts, and then push the resolved code back to the remote repository.

Partitioning is effective for low-cardinality columns like date. For a high-cardinality column like product_category, adding it as a partition key would create too many small files (the small files problem) and be inefficient. Z-Ordering is the correct technique here. It co-locates related data from different columns (in this case, product_category) within the same set of files. This allows the query engine to skip a significant amount of data that doesn't match the filter conditions, dramatically improving query performance without the downsides of fine-grained partitioning.

Lakehouse Federation is the feature designed for this exact use case. It allows you to create read-only connections to external database systems like PostgreSQL and register their tables as foreign tables within Unity Catalog. This approach securely manages credentials using Databricks Secrets and allows users to run standard SQL queries that join data across the Databricks and PostgreSQL systems seamlessly and efficiently, pushing down predicates where possible.

This statement is false. A key feature of Databricks Asset Bundles is the use of 'targets' in the databricks.yml configuration file. Targets allow a single bundle to define deployment configurations for multiple environments, such as development, staging, and production, which can correspond to different Databricks workspaces. This makes bundles a powerful tool for managing deployments across a project's entire lifecycle.