Row Filters

Read this if you use Streaming Tables in Lakeflow Spark Declarative Pipelines

🚀 We’re excited to announce that Lakeflow Spark Declarative Pipelines (SDP) now supports creating “vanilla” (i.e., non STREAMING) MANAGED TABLES and writing to them via one or more append flows , using the new CREATE TABLE ... FLOW ( SQL ) and create_table() (Python) APIs . What is this Beta? This Beta allows creating a managed table that is populated by append flows: CREATE TABLE ... FLOW (SQL) / create_table() + @append_flow (Python) create a managed table written by one or more flows. Fan multiple sources into one table — declare several flows targeting the same managed table. Full table surface works: partitioning, liquid clustering, expectations, row filters, table properties, and private (pipeline-local) tables. import_checkpoint on append_flow , which migrates an existing Structured Streaming workload into a pipeline without reprocessing the source — the flow imports the query's existing checkpoint and resumes from the last committed offset with state intact. Example (Python): from pyspark import pipelines as dp dp.create_table("combined") dp.append_flow(target="combined") def from_a(): return spark.readStream.table("source_a") u/dp.append_flow(target="combined") def from_b(): return spark.readStream.table("source_b") Example (SQL): CREATE TABLE events PARTITIONED BY (bucket) FLOW INSERT BY NAME SELECT id, bucket FROM STREAM read_files('abfss://my_path', format => 'json'); Where do we need help? We are in Beta, so there might be some rough edges. Please take this for a spin and share your feedback here . What’s next? Managed Tables support for other flow types (AutoCDC, Replace Using, and Replace Where) is coming soon! Learn more CREATE TABLE ... FLOW (SQL reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-table-flow create_table (Python reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-create-table import_checkpoint on append_flow — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-append-flow Questions, feedback, or help: comment below or share feedback in the form: https://forms.gle/7bGP5FYN7P1Z4WP27 submitted by /u/SlightImagination250 [link] [comments]

Ingest image in databricks for powerbi ? a poc and any idea welcome

Spent some time this week on a POC that started from a business constraint: about 3,000 photos to ingest every week, sensitive enough that we can't just drop a shareable link in a dashboard, and they need to end up in Power BI where people already work, with row-level security. That combination rules out the easy answer. No public links, no loose files in blob storage floating around outside governance. The images had to live inside Delta on Databricks so Unity Catalog could handle access control, and Power BI had to be able to render them directly from the table. What the simple poc below does: Reads the images with Spark's binaryFile source (recursive lookup, glob filter on *.jpg) to pull path, modification time, size and raw bytes into one DataFrame. Encodes the binary content as a base64 data URL, so the image itself lives inside the row instead of behind a link. Then the actual blocker: Power BI caps text fields at roughly 32,766 characters, and a real photo's base64 string blows straight past that. So each string gets split into ~32,000-character segments and exploded into multiple rows, each tagged with its index and total length. On the Power BI side, a single DAX measure puts it back together in the right order before rendering: ​ Image_concat = IF( HASONEVALUE(images_in_delta[image_name]), CONCATENATEX( images_in_delta, images_in_delta[segment], , images_in_delta[split_index] ) ) Not elegant, but it's what gets a full-resolution image through a hard platform limit without touching the sensitivity requirement. The PySpark side, stripped to what matters — reading the images and doing the chunking: from pyspark.sql import functions as F from pyspark.sql import DataFrame #READ ALL THE IMAGES images_df = spark.read.format("binaryFile") \ .option("recursiveFileLookup", "true") \ .option("pathGlobFilter", "*.jpg") \ .load("/Volumes/main/image_ingest/image_sample") def add_base64url_from_image_binary(df: DataFrame, max_len: int = 32000) -> DataFrame: df_with_b64 = df.select( "*", F.concat(F.lit("data:image/jpg;base64,"), F.base64(F.col("content"))).alias("base64url") ) df_with_split_info = df_with_b64.select( "*", F.ceil(F.length(F.col("base64url")) / F.lit(max_len)).cast("int").alias("num_segments"), F.length(F.col("base64url")).alias("total_length") ) df_split = ( df_with_split_info .withColumn( "split_index", F.explode(F.sequence(F.lit(0), F.col("num_segments") - 1)) ) .select( "*", F.substring( F.col("base64url"), F.col("split_index") * max_len + 1, F.least(F.lit(max_len), F.col("total_length") - F.col("split_index") * max_len) ).alias("segment") ) .drop("base64url", "content") ) return df_split def add_image_name(df : DataFrame) -> DataFrame : return df.withColumn("image_name", F.regexp_replace(F.col("path"),".*/([^/]+)$", "$1")) df_images = add_base64url_from_image_binary(images_df) df_images = add_image_name(df_images) df_images.write.mode("overwrite").format("delta") \ .option("mergeSchema", "true") \ .saveAsTable("main.image_ingest.images_in_delta") Nothing here is exotic engineering — the interesting part was realizing early that the constraint wasn't really "how do we store images in Delta," it was "how do we get a sensitive image through Power BI's text field limit without ever exposing it outside the governed table." Once that was clear, the chunking workaround fell out naturally. At 3,000 images a week this holds up. If volume goes up meaningfully, I'd want to revisit whether inlining every image is still the right call versus resolving binary content on demand. Curious if others have hit the same Power BI ceiling with sensitive image data and landed on something cleaner than manual chunking. Have you any other idea than this ? submitted by /u/Data-space_men [link] [comments]

Your Row Filter Works. The Agent Just Isn't Who It's Filtering

What is the recommended approach to enforce row-level security in Unity Catalog for external BI tool

Databricks Data Engineer Associate Exam Updated for 2026

The Databricks Data Engineer Associate exam changed on May 4, 2026. The exam now has 7 domains instead of 5. Two new domains were added. The first new domain is CI/CD. This includes: • Databricks Repos • Git integration • Branching and commits • Deploying Declarative Automation Bundles • Using the Databricks CLI • Moving code from dev to test to production Databricks Asset Bundles is now called Declarative Automation Bundles, so learn the new name. If you have never used Git or the Databricks CLI inside Databricks, spend some time practicing in the Free Edition. Connect a Git repo, make commits, and deploy bundles. Hands-on practice will help a lot. The second new domain is Troubleshooting, Monitoring, and Optimization. This includes: • Reading the Spark UI • Finding bottlenecks like data skew and excessive shuffling • Understanding Liquid Clustering • Predictive optimization • Troubleshooting cluster and memory issues Many courses do not teach Spark UI deeply, so try running queries yourself and checking the Spark UI. Compare good queries with inefficient ones to understand the difference. Some existing domains also changed. Ingestion now includes Lakeflow Connect along with Auto Loader and COPY INTO. Governance now includes: • Column-level masking • Row-level security • Attribute-based access control You now need to understand security beyond basic GRANT permissions. Lakeflow Jobs also tests three trigger types: • Scheduled • File arrival • Table update Know when to use each one. Some product names also changed: • Databricks Asset Bundles → Declarative Automation Bundles • Delta Live Tables → Lakeflow Declarative Pipelines The exam uses the new terminology, so update your study material if you are using older resources. The exam format is still: • 45 scored questions • 90 minutes • $200 There may also be extra unscored questions mixed into the exam. For preparation, the original Academy courses still help for the old domains. But for the two new domains, hands-on practice is very important. Practice: • Spark UI • Git integration • Databricks CLI • Deployments using bundles Also read the latest official exam guide PDF from the Databricks page. Good luck to everyone preparing for the exam.