How to
How to run and schedule Power BI report tests, review visual changes, set thresholds and notifications, and manage members.
Run a test package
Select Run now on the package page, or Run next to the package on its suite page. The run is queued and starts when a worker is free; runs of different packages execute in parallel. Follow it under Running now on the overview or under In progress on Runs.
Schedule a test package
- 1Open Schedules and select New schedule, or Add schedule on the package page.
- 2Choose the test package, daily or weekly, and the time in UTC.
- 3Add schedule. The switch next to it pauses and resumes it.
Review a visual change
When a page looks different from its baseline, its visual result is Changed and the run lists it as a change to review. Changes alone do not fail a run.
- 1Open the run and choose the changed result.
- 2Compare the screenshots under Visual review.
- 3Approve as new baseline if the change is expected: later runs compare with the new screenshot. Keep baseline if it is not: the old screenshot stays, and you may want to Create issue.
Change the pages and checks
On the package page, Edit opens the wizard with the current pages and checks; Save changes replaces them, while the suite, schedules and run history stay. To skip a check on one page only, turn it off in the table on the package page and save.
Move or delete a test package
Move on the package page puts it in another suite. Delete removes the package and its schedules; its runs stay in the history with their evidence. Deleting a suite moves its packages to the Trial suite.
Track a problem as an issue
On a run page, select Create issue on the result. The title and severity are filled in from the check; change them if needed and create the issue. Find it under Issues, and resolve it once the report is fixed.
Add the organization connection
Admins and owners can add a service principal that every member can save packages with, and that reads the Power BI inventory:
- 1In Microsoft Entra ID, register an app and create a client secret. Do not add Power BI API permissions to it: the service principal gets its access from the Power BI settings below.
- 2In the Power BI admin portal, under Tenant settings, enable Service principals can use Fabric APIs. For the Scanner API (see Power BI inventory), also enable Service principals can access read-only admin APIs; it applies only to security groups, so add the app to a security group and choose that group. Settings take up to 15 minutes to apply.
- 3Add the app, or a security group it is in, to each workspace to test (in Power BI, Manage access, as Member or Admin). It sees only those workspaces, never anyone's My workspace.
- 4In Settings › Organization, enter the Tenant ID, Client ID and Client secret and choose the Cloud. For reports with row-level security, enter the Effective username to view them as.
- 5Save. The connection is verified with Microsoft before it is stored, and the page shows how many workspaces it sees, or what to change when it sees none; the secret is encrypted and never shown again. After adding it to more workspaces, select Verify again.
Put the workspaces to test on a Premium or Fabric capacity (a Fabric trial is enough). In other workspaces, Power BI lets a service principal show reports only a limited number of times, for development, and marks them This is a free trial version, on screenshots too. Microsoft's sample reports installed from Learn cannot be read by a service principal at all: upload them as .pbix files.
Get notified when runs finish
Admins open Settings › Notifications, enter a Webhook URL, choose which results to send (Failed, Changed, Warning, Passed) and turn notifications on. Send test sends a test message right away, Remove webhook forgets the address and turns notifications off. Recent deliveries shows what was sent, and why a run was skipped.
Each message is JSON with an event (run-finished, powerbi-reconnect-required or test) and a text line that chat tools show as it is. A finished run also names its package and suite, its result, the checks that failed, warned or changed, and links to the run.
Saving a webhook URL shows its signing secret once. Every request carries X-Power-BI-Lab-Timestamp (Unix seconds) and X-Power-BI-Lab-Signature: v1= followed by the hex HMAC-SHA256, keyed with the secret, of the timestamp, a dot and the raw body. Your endpoint computes the same, compares, and refuses timestamps older than a few minutes. Saving another URL replaces the secret.
The webhook must be on a public address: addresses of a private network or of the server itself are refused, and redirects are not followed. A webhook that is busy, failing or does not answer within 10 seconds is tried up to three times.
Change check thresholds
In Settings › Organization, under Check thresholds, admins set:
- Render and data age
- When a passing check becomes a warning: a page that takes longer than 30 seconds to render, or data older than 24 hours, by default.
- Screenshot matches up to
- The share of pixels a screenshot may differ from its baseline and still match it, 0.1% by default. Power BI never draws a page exactly the same twice, so smaller differences are noise; 0 makes any difference a change.
- Unstable screenshot and fails often
- The share of a page's recent runs from which Trends flags its screenshot as unstable (30%) or the page as failing often (20%).
Manage members and roles
In Settings › Members, change a member's role or remove them. A removed member's sessions end and signing in again does not bring them back; you can restore them from the same list. Admins manage members and viewers; owners also manage admins and owners. An organization always keeps one owner. To bring in someone from another Microsoft organization, enter their email and a role under Invitations and send them the link: it is shown once, and only the person whose Microsoft account has that email can use it, once. Revoke stops a link that has not been used.
Download the evidence of a run
Download evidence on a run page saves a JSON file with the run, the fingerprint of the package that ran, every step with its evidence, the log and the links to its screenshots.