Reference
The checks a Power BI Lab run makes, results and run statuses, roles, and a glossary of the terms the console uses.
Checks
Report access and page discovery run in every run; the other checks run on the pages they are chosen for. Each run also records the lineage of its reports from the Power BI inventory.
- Report access
- The report opens with the package connection.
- Page discovery
- Each tested page still exists in the report; a deleted or renamed page fails.
- Visual regression
- The page screenshot is compared with its approved baseline; a difference larger than the visual tolerance waits for review (see Change check thresholds). The screenshot is taken once the page stops changing, so visuals that are still drawing do not count as changes.
- Data freshness
- The datasets behind the report refreshed recently; older than the threshold is a warning.
- Accessibility
- Controls have labels and the page has headings and landmarks.
- Performance
- The page renders within the threshold; slower is a warning.
Results and run statuses
- Passed
- The check found nothing.
- Warning
- Worth a look, for example stale data or a slow page; the run still passes.
- Changed
- A visual difference that waits for someone to approve or keep the baseline; the run still passes.
- Failed
- The check found a problem, or could not reach the page; the run fails.
- Skipped
- Not checked, for example a first visual check without a baseline.
A run is Queued until a worker is free, Running while it checks pages, then Passed or Failed.
Roles
- Viewer
- Sees everything in the organization and connects their own Power BI.
- Member
- Also creates and changes test packages and suites, runs them, schedules them, decides on visual changes, tracks issues, refreshes the Power BI inventory and dismisses changes and suggestions.
- Admin
- Also manages the organization connection, notifications, thresholds, the daily inventory sync time, and members and viewers.
- Owner
- Also manages admins and owners.
Glossary
The words the console and this documentation use, in alphabetical order.
- Audit log
- The record of who did what in the organization: saving packages, starting runs, changing connections, roles and settings.
- Baseline
- The approved screenshot of a report page. Visual checks compare every new screenshot of the page with it; earlier baselines are kept as versions.
- Change in Power BI
- Something a sync of the Power BI inventory found different from the sync before, such as a removed page or a dataset reading other data sources, with the test packages it affects (see Changes in Power BI).
- Check
- One thing a run verifies on a page or report: report access, page discovery, visual regression, data freshness, accessibility or performance (see Checks).
- Connect as me
- Your own Power BI sign-in, used by the test packages you save, scheduled runs included. Reports in My workspace can only be tested with it.
- Coverage
- Which report pages test packages check, and the latest result of each configured check.
- Dataset
- The Power BI semantic model a report reads its data from; data freshness looks at when it last refreshed.
- Effective username
- The user whose row-level security the organization connection views reports as.
- Evidence
- What a run records to show what it found: each step with its details, screenshots, timings and the log. Download it as JSON from the run page.
- Fingerprint
- A SHA-256 hash of the exact test package a run executed, so a run can be traced to the package as it was then, even after the package changes.
- Insights
- The console page with the Power BI inventory, changes in Power BI, suggestions and trends over the recent runs.
- Inventory
- What the organization connection sees in Power BI: workspaces, reports and pages, datasets, dataflows, data sources and what depends on what. It is read daily and on demand.
- Issue
- A problem found in a run that someone follows up until it is resolved; high or warning severity.
- Key
- The short code at the end of a suite or test package address, such as
a3f2b1c4in/app/suites/finance-regression-a3f2b1c4. The readable name before it may change; the key stays. - Lineage
- Where a report's data comes from: its dataset, the dataflows and data sources behind it, and the other reports that use the same dataset.
- Member
- A person with access to the organization, with one of four roles: viewer, member, admin or owner (see Roles).
- Organization
- Everyone who signs in from the same Microsoft organization (Microsoft Entra tenant) works in one Power BI Lab organization, with shared suites, runs and settings. People from other Microsoft organizations join by invitation; someone in several organizations switches between them in the account menu.
- Organization connection
- A service principal added by an admin, so test packages do not depend on one person's Power BI sign-in.
- Page
- A page of a Power BI report. A test package checks the pages chosen for it, each with its own set of checks.
- Render time
- How long a report page takes to show completely; above the threshold, the performance check warns.
- Report
- A Power BI report in a workspace; a test package checks one report.
- Result
- The outcome of one check on one page in a run: passed, warning, changed, failed or skipped.
- Run
- One execution of a test package, manual or scheduled, with its results and evidence. Runs are numbered per organization: #1, #2 …
- Scanner API
- A Power BI administration interface that adds change dates, data sources and lineage to the inventory, where the Power BI administrator allows the organization connection to use it.
- Schedule
- When a test package runs on its own: daily or weekly at a time in UTC. A run missed while no worker was free is caught up once.
- Service principal
- An app registered in Microsoft Entra ID that signs in to Power BI as itself, with a client ID and secret instead of a person's account.
- Suggestion
- Something Insights proposes to test next, such as a report without a package, with the action that resolves it.
- Suite
- A folder of test packages, for example per team or business area. Packages of a deleted suite move to the Trial suite.
- Test package
- One report, the pages to test and the checks to run on each page, with the Power BI access it runs with. Packages run manually or on schedules.
- Threshold
- A limit admins set for the organization: when a passing check becomes a warning (render time, data age), how much a screenshot may differ and still match, and when Insights flags a page.
- Trial suite
- The suite every organization starts with. It cannot be deleted; packages of deleted suites move to it.
- Trigger
- What started a run: a person (manual) or a schedule.
- Visual change
- A difference between a page's screenshot and its baseline. It waits for someone to approve it as the new baseline or keep the old one; it does not fail the run.
- Visual tolerance
- The share of pixels a screenshot may differ from its baseline and still match it, 0.1% by default; smaller differences are how Power BI draws a page slightly differently each time.
- Webhook
- An address of your own that Power BI Lab calls when a run finishes, for example to post to a chat channel.
- Worker
- The process that executes runs. Queued runs wait for a free worker; several runs execute in parallel.
- Workspace
- A Power BI workspace holding reports and datasets; My workspace is your personal one.