Power BI LabReport assurance
Contents
Documentation

Reference

The checks a Power BI Lab run makes, results and run statuses, roles, and a glossary of the terms the console uses.

Checks

Report access and page discovery run in every run; the other checks run on the pages they are chosen for. Each run also records the lineage of its reports from the Power BI inventory.

Report access
The report opens with the package connection.
Page discovery
Each tested page still exists in the report; a deleted or renamed page fails.
Visual regression
The page screenshot is compared with its approved baseline; a difference larger than the visual tolerance waits for review (see Change check thresholds). The screenshot is taken once the page stops changing, so visuals that are still drawing do not count as changes.
Data freshness
The datasets behind the report refreshed recently; older than the threshold is a warning.
Accessibility
Controls have labels and the page has headings and landmarks.
Performance
The page renders within the threshold; slower is a warning.

Results and run statuses

Passed
The check found nothing.
Warning
Worth a look, for example stale data or a slow page; the run still passes.
Changed
A visual difference that waits for someone to approve or keep the baseline; the run still passes.
Failed
The check found a problem, or could not reach the page; the run fails.
Skipped
Not checked, for example a first visual check without a baseline.

A run is Queued until a worker is free, Running while it checks pages, then Passed or Failed.

Roles

Viewer
Sees everything in the organization and connects their own Power BI.
Member
Also creates and changes test packages and suites, runs them, schedules them, decides on visual changes, tracks issues, refreshes the Power BI inventory and dismisses changes and suggestions.
Admin
Also manages the organization connection, notifications, thresholds, the daily inventory sync time, and members and viewers.
Owner
Also manages admins and owners.

Glossary

The words the console and this documentation use, in alphabetical order.

Audit log
The record of who did what in the organization: saving packages, starting runs, changing connections, roles and settings.
Baseline
The approved screenshot of a report page. Visual checks compare every new screenshot of the page with it; earlier baselines are kept as versions.
Change in Power BI
Something a sync of the Power BI inventory found different from the sync before, such as a removed page or a dataset reading other data sources, with the test packages it affects (see Changes in Power BI).
Check
One thing a run verifies on a page or report: report access, page discovery, visual regression, data freshness, accessibility or performance (see Checks).
Connect as me
Your own Power BI sign-in, used by the test packages you save, scheduled runs included. Reports in My workspace can only be tested with it.
Coverage
Which report pages test packages check, and the latest result of each configured check.
Dataset
The Power BI semantic model a report reads its data from; data freshness looks at when it last refreshed.
Effective username
The user whose row-level security the organization connection views reports as.
Evidence
What a run records to show what it found: each step with its details, screenshots, timings and the log. Download it as JSON from the run page.
Fingerprint
A SHA-256 hash of the exact test package a run executed, so a run can be traced to the package as it was then, even after the package changes.
Insights
The console page with the Power BI inventory, changes in Power BI, suggestions and trends over the recent runs.
Inventory
What the organization connection sees in Power BI: workspaces, reports and pages, datasets, dataflows, data sources and what depends on what. It is read daily and on demand.
Issue
A problem found in a run that someone follows up until it is resolved; high or warning severity.
Key
The short code at the end of a suite or test package address, such as a3f2b1c4 in /app/suites/finance-regression-a3f2b1c4. The readable name before it may change; the key stays.
Lineage
Where a report's data comes from: its dataset, the dataflows and data sources behind it, and the other reports that use the same dataset.
Member
A person with access to the organization, with one of four roles: viewer, member, admin or owner (see Roles).
Organization
Everyone who signs in from the same Microsoft organization (Microsoft Entra tenant) works in one Power BI Lab organization, with shared suites, runs and settings. People from other Microsoft organizations join by invitation; someone in several organizations switches between them in the account menu.
Organization connection
A service principal added by an admin, so test packages do not depend on one person's Power BI sign-in.
Page
A page of a Power BI report. A test package checks the pages chosen for it, each with its own set of checks.
Render time
How long a report page takes to show completely; above the threshold, the performance check warns.
Report
A Power BI report in a workspace; a test package checks one report.
Result
The outcome of one check on one page in a run: passed, warning, changed, failed or skipped.
Run
One execution of a test package, manual or scheduled, with its results and evidence. Runs are numbered per organization: #1, #2 …
Scanner API
A Power BI administration interface that adds change dates, data sources and lineage to the inventory, where the Power BI administrator allows the organization connection to use it.
Schedule
When a test package runs on its own: daily or weekly at a time in UTC. A run missed while no worker was free is caught up once.
Service principal
An app registered in Microsoft Entra ID that signs in to Power BI as itself, with a client ID and secret instead of a person's account.
Suggestion
Something Insights proposes to test next, such as a report without a package, with the action that resolves it.
Suite
A folder of test packages, for example per team or business area. Packages of a deleted suite move to the Trial suite.
Test package
One report, the pages to test and the checks to run on each page, with the Power BI access it runs with. Packages run manually or on schedules.
Threshold
A limit admins set for the organization: when a passing check becomes a warning (render time, data age), how much a screenshot may differ and still match, and when Insights flags a page.
Trial suite
The suite every organization starts with. It cannot be deleted; packages of deleted suites move to it.
Trigger
What started a run: a person (manual) or a schedule.
Visual change
A difference between a page's screenshot and its baseline. It waits for someone to approve it as the new baseline or keep the old one; it does not fail the run.
Visual tolerance
The share of pixels a screenshot may differ from its baseline and still match it, 0.1% by default; smaller differences are how Power BI draws a page slightly differently each time.
Webhook
An address of your own that Power BI Lab calls when a run finishes, for example to post to a chat channel.
Worker
The process that executes runs. Queued runs wait for a free worker; several runs execute in parallel.
Workspace
A Power BI workspace holding reports and datasets; My workspace is your personal one.