Row Filters

Row-level security for a RAG agent: what Unity Catalog enforces, and what you have to build

Ontology ranked the snippet. I wrote the gate I wanted before trusting a Genie space.

A commercial user asked what yield was last season. Two definitions live in the same lakehouse. One is plot-level, moisture-adjusted, borders out. The other is as-harvested, and a dashboard still uses it. Genie returned the second number. Not a hallucination. OntoRank did what it does: creator, usage, link to a certified asset, freshness. The loud dashboard won. Nobody had certified the measure the board pack is allowed to quote. That is the failure I care about. A green demo is not a release gate. What I actually require before that space faces a user who will paste the sentence into Slack: Golden questions written before the space exists. If you write them after you have tuned instructions, you are scoring an overfit prompt. Benchmarks are not instructions. Genie does not learn from the benchmark SQL. The number lives in a Metric View. A column comment and a knowledge-store formula are hints. Ontology will rank all three. Only the Metric View is a contract every surface compiles. Same KPI from MEASURE(), from the certified dashboard, and from Genie. If they disagree, the space stays off Slack. Permissions checked as the asker. Inside a Genie Agent, compute is the author’s warehouse and data access is the end user. That split does not automatically hold in Teams, Copilot on maker credentials, or a Databricks App on its default service principal. The row filter still runs. It runs for whoever showed up on the SQL. A written kill line. I use Databricks’ “above 80% before UAT” as the hold line, not the ship line. Ship is 90% and zero fails on a regulated definition. Any miss on that definition takes the space down. A person does that. Not a service principal. Genie Code can draft the Metric View and the MERGE. ZeroOps can propose a fix and wait for Approve. Neither owns the definition. If the job stays green for eleven days because a key was slightly wrong, that is still your incident. I wrote the four pieces up so I would stop re-explaining this in reviews. Argue the 80 vs 90 line if you want. I will not move a regulated KPI to “looks fine in the demo.” submitted by /u/Worth_Phase1358 [link] [comments]

Row-level security when the filter column is not on your table

Managing Row-Level Security (RLS) vs. Views: Performance impacts at scale

Best practice to enforcing row-level security across multiple catalogs sharing same schema structure

Read this if you use Streaming Tables in Lakeflow Spark Declarative Pipelines

🚀 We’re excited to announce that Lakeflow Spark Declarative Pipelines (SDP) now supports creating “vanilla” (i.e., non STREAMING) MANAGED TABLES and writing to them via one or more append flows , using the new CREATE TABLE ... FLOW ( SQL ) and create_table() (Python) APIs . What is this Beta? This Beta allows creating a managed table that is populated by append flows: CREATE TABLE ... FLOW (SQL) / create_table() + @append_flow (Python) create a managed table written by one or more flows. Fan multiple sources into one table — declare several flows targeting the same managed table. Full table surface works: partitioning, liquid clustering, expectations, row filters, table properties, and private (pipeline-local) tables. import_checkpoint on append_flow , which migrates an existing Structured Streaming workload into a pipeline without reprocessing the source — the flow imports the query's existing checkpoint and resumes from the last committed offset with state intact. Example (Python): from pyspark import pipelines as dp dp.create_table("combined") dp.append_flow(target="combined") def from_a(): return spark.readStream.table("source_a") u/dp.append_flow(target="combined") def from_b(): return spark.readStream.table("source_b") Example (SQL): CREATE TABLE events PARTITIONED BY (bucket) FLOW INSERT BY NAME SELECT id, bucket FROM STREAM read_files('abfss://my_path', format => 'json'); Where do we need help? We are in Beta, so there might be some rough edges. Please take this for a spin and share your feedback here . What’s next? Managed Tables support for other flow types (AutoCDC, Replace Using, and Replace Where) is coming soon! Learn more CREATE TABLE ... FLOW (SQL reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-table-flow create_table (Python reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-create-table import_checkpoint on append_flow — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-append-flow Questions, feedback, or help: comment below or share feedback in the form: https://forms.gle/7bGP5FYN7P1Z4WP27 submitted by /u/SlightImagination250 [link] [comments]

Ingest image in databricks for powerbi ? a poc and any idea welcome

Spent some time this week on a POC that started from a business constraint: about 3,000 photos to ingest every week, sensitive enough that we can't just drop a shareable link in a dashboard, and they need to end up in Power BI where people already work, with row-level security. That combination rules out the easy answer. No public links, no loose files in blob storage floating around outside governance. The images had to live inside Delta on Databricks so Unity Catalog could handle access control, and Power BI had to be able to render them directly from the table. What the simple poc below does: Reads the images with Spark's binaryFile source (recursive lookup, glob filter on *.jpg) to pull path, modification time, size and raw bytes into one DataFrame. Encodes the binary content as a base64 data URL, so the image itself lives inside the row instead of behind a link. Then the actual blocker: Power BI caps text fields at roughly 32,766 characters, and a real photo's base64 string blows straight past that. So each string gets split into ~32,000-character segments and exploded into multiple rows, each tagged with its index and total length. On the Power BI side, a single DAX measure puts it back together in the right order before rendering: ​ Image_concat = IF( HASONEVALUE(images_in_delta[image_name]), CONCATENATEX( images_in_delta, images_in_delta[segment], , images_in_delta[split_index] ) ) Not elegant, but it's what gets a full-resolution image through a hard platform limit without touching the sensitivity requirement. The PySpark side, stripped to what matters — reading the images and doing the chunking: from pyspark.sql import functions as F from pyspark.sql import DataFrame #READ ALL THE IMAGES images_df = spark.read.format("binaryFile") \ .option("recursiveFileLookup", "true") \ .option("pathGlobFilter", "*.jpg") \ .load("/Volumes/main/image_ingest/image_sample") def add_base64url_from_image_binary(df: DataFrame, max_len: int = 32000) -> DataFrame: df_with_b64 = df.select( "*", F.concat(F.lit("data:image/jpg;base64,"), F.base64(F.col("content"))).alias("base64url") ) df_with_split_info = df_with_b64.select( "*", F.ceil(F.length(F.col("base64url")) / F.lit(max_len)).cast("int").alias("num_segments"), F.length(F.col("base64url")).alias("total_length") ) df_split = ( df_with_split_info .withColumn( "split_index", F.explode(F.sequence(F.lit(0), F.col("num_segments") - 1)) ) .select( "*", F.substring( F.col("base64url"), F.col("split_index") * max_len + 1, F.least(F.lit(max_len), F.col("total_length") - F.col("split_index") * max_len) ).alias("segment") ) .drop("base64url", "content") ) return df_split def add_image_name(df : DataFrame) -> DataFrame : return df.withColumn("image_name", F.regexp_replace(F.col("path"),".*/([^/]+)$", "$1")) df_images = add_base64url_from_image_binary(images_df) df_images = add_image_name(df_images) df_images.write.mode("overwrite").format("delta") \ .option("mergeSchema", "true") \ .saveAsTable("main.image_ingest.images_in_delta") Nothing here is exotic engineering — the interesting part was realizing early that the constraint wasn't really "how do we store images in Delta," it was "how do we get a sensitive image through Power BI's text field limit without ever exposing it outside the governed table." Once that was clear, the chunking workaround fell out naturally. At 3,000 images a week this holds up. If volume goes up meaningfully, I'd want to revisit whether inlining every image is still the right call versus resolving binary content on demand. Curious if others have hit the same Power BI ceiling with sensitive image data and landed on something cleaner than manual chunking. Have you any other idea than this ? submitted by /u/Data-space_men [link] [comments]

Your Row Filter Works. The Agent Just Isn't Who It's Filtering

What is the recommended approach to enforce row-level security in Unity Catalog for external BI tool

Databricks Data Engineer Associate Exam Updated for 2026

The Databricks Data Engineer Associate exam changed on May 4, 2026. The exam now has 7 domains instead of 5. Two new domains were added. The first new domain is CI/CD. This includes: • Databricks Repos • Git integration • Branching and commits • Deploying Declarative Automation Bundles • Using the Databricks CLI • Moving code from dev to test to production Databricks Asset Bundles is now called Declarative Automation Bundles, so learn the new name. If you have never used Git or the Databricks CLI inside Databricks, spend some time practicing in the Free Edition. Connect a Git repo, make commits, and deploy bundles. Hands-on practice will help a lot. The second new domain is Troubleshooting, Monitoring, and Optimization. This includes: • Reading the Spark UI • Finding bottlenecks like data skew and excessive shuffling • Understanding Liquid Clustering • Predictive optimization • Troubleshooting cluster and memory issues Many courses do not teach Spark UI deeply, so try running queries yourself and checking the Spark UI. Compare good queries with inefficient ones to understand the difference. Some existing domains also changed. Ingestion now includes Lakeflow Connect along with Auto Loader and COPY INTO. Governance now includes: • Column-level masking • Row-level security • Attribute-based access control You now need to understand security beyond basic GRANT permissions. Lakeflow Jobs also tests three trigger types: • Scheduled • File arrival • Table update Know when to use each one. Some product names also changed: • Databricks Asset Bundles → Declarative Automation Bundles • Delta Live Tables → Lakeflow Declarative Pipelines The exam uses the new terminology, so update your study material if you are using older resources. The exam format is still: • 45 scored questions • 90 minutes • $200 There may also be extra unscored questions mixed into the exam. For preparation, the original Academy courses still help for the old domains. But for the two new domains, hands-on practice is very important. Practice: • Spark UI • Git integration • Databricks CLI • Deployments using bundles Also read the latest official exam guide PDF from the Databricks page. Good luck to everyone preparing for the exam.