Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions doc/gui/0_gui.md
Original file line number Diff line number Diff line change
Expand Up @@ -722,6 +722,63 @@ If an attack no longer exists, its entry stays, shows the attack's ID and
attachment time, and reads **Evidence unavailable**. **Remove link** removes only
the link, never the attack. Deleting a finding removes its links the same way.

#### Exporting findings

**Export findings** on an operation page downloads the findings you select as
one document to copy into a larger report. The dialog lists every finding in the
operation, not only the 20 on the current page, and starts with all of them
selected. Clear the findings to leave out, or use **Select all** to select or
clear the whole list; it shows a mixed state when only some are selected. A
finding added after you open the dialog appears the next time you open it.

Under **Format**, choose **Word (.docx)**, the default, or **Markdown (.md)**, then
**Download Word** or **Download Markdown**. Both formats hold the same findings,
fields, and evidence. Word puts every finding in one table with **Finding** and
**Assessment** columns, so you can copy rows into your report. Markdown lists
the same information from top to bottom.
PyRIT doesn't use report templates and never uploads or reads your report.
Files are named after the operation and your computer's local time, for example
`copyrit-findings-synthetic-evaluation-2026-10-22-1258.docx`.

The document is headed with the operation's name, and findings follow the
dashboard's order: most severe first, newest first within each level. The
**Finding** column holds each title, description, and evidence; **Assessment**
holds severity and harm-type. Markdown puts severity and harm-type first under
each title. A Word row with synthetic content reads:

```text
Finding | Assessment
--------------------------------------------------+------------------------------------
Restricted test marker disclosed | Severity: Important
Description: The assistant returned a synthetic | Harm-type: Proprietary, Confidential
marker that the test policy instructed it to | & Classified Information
withhold. |
Evidence: |
• Restricted-marker test (link) |
Outcome: Success · Score: True (SubStringScorer)|
Rationale: The response contained the |
restricted marker. |
```

Each attached attack is a bullet led by its objective, which links to the same
conversation as **Open attack**. Under it come the outcome, the same score as
the evidence table (with **human review** as the scorer when a person scored it,
or **Not scored**), the score's rationale, and the attack's operation when it
isn't this one. An attack that no longer exists keeps its entry under its attack
ID and reads **Evidence unavailable**; a finding without evidence reads
**Evidence: none attached**. Messages, attachments, and media are never
included. The document reads fine offline, but its links open only for someone
with access to this PyRIT deployment.

Each download reads the findings and evidence as saved at that moment, and its
links open the live attacks, which can gain messages and scores later. If a
selected finding is deleted before you download, the download fails.

> **Note:** Findings, attack objectives, and score rationales can contain
> sensitive content, and nothing is redacted automatically. Your PyRIT server
> generates the document and sends it to no other service. Store and share
> exported files responsibly.

#### Upgrading an existing database

One migration adds Operations, findings, evidence, and optional Operation IDs on
Expand Down
160 changes: 159 additions & 1 deletion frontend/e2e/operations.spec.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
import { readFileSync } from 'node:fs'

import { test, expect } from '@playwright/test'

import type { Finding, Operation } from '../src/types'
import type {
CreateAttackResponse, Finding, FindingEvidenceAttachResponse, FindingExportFormat, Operation,
} from '../src/types'
import { waitForAppReady } from './_runtime'

for (const width of [1280, 390]) {
Expand Down Expand Up @@ -351,3 +355,157 @@ test('selects saved operations inline and revisits ID-scoped attack history @see
await expect(historyLink).toBeVisible()
await expect(page.getByRole('heading', { name: 'Independent human finding', exact: true })).toBeVisible()
})

test('exports selected findings as Word and Markdown @seeded', async ({ page, request }) => {
test.setTimeout(90_000)
const version = await (await request.get('/api/version')).json()
const headers = { 'PyRIT-Compatibility-ID': version.compatibility_id }
const operationResponse = await request.post('/api/operations', {
headers, data: { name: `Findings export ${Date.now()}` },
})
expect(operationResponse.status()).toBe(201)
const operation: Operation = await operationResponse.json()
const description = 'Ünïcødé 😀 中文 first line\n```\n# Not a finding\n<b>kept literal</b>\nlast line'
const longTitle = `Beyond the first page ${'unbroken'.repeat(30)}`
// Saved out of severity order, so the document's order can only come from severity.
const seeds = [
{ title: 'Export low', severity: 'low' },
{ title: 'Export critical', severity: 'critical', description },
{ title: longTitle, severity: 'informational' },
...Array.from({ length: 17 }, (_, index) => ({
title: `Export moderate ${String(index).padStart(2, '0')}`, severity: 'moderate',
})),
{ title: 'Export important', severity: 'important' },
]
const findings = new Map<string, Finding>()
for (const seed of seeds) {
const response = await request.post(`/api/operations/${operation.id}/findings`, { headers, data: seed })
expect(response.status()).toBe(201)
findings.set(seed.title, await response.json())
}
const findingId = (title: string): string => {
const finding = findings.get(title)
if (!finding) throw new Error(`Expected a saved finding titled ${title}.`)
return finding.id
}
const attackResponse = await request.post('/api/attacks', {
headers, data: {
target_registry_name: 'browser_offline',
operation_id: operation.id,
prepended_conversation: [{ role: 'user', pieces: [{ original_value: 'Saved offline evidence.' }] }],
},
})
expect(attackResponse.ok()).toBeTruthy()
const attack: CreateAttackResponse = await attackResponse.json()
const attachResponse = await request.post(
`/api/operations/${operation.id}/findings/${findingId('Export critical')}/evidence`,
{ headers, data: { attack_result_id: attack.attack_result_id } },
)
expect(attachResponse.ok()).toBeTruthy()
const evidence: FindingEvidenceAttachResponse = await attachResponse.json()
const selectedTitles = ['Export critical', 'Export moderate 03', 'Export low', longTitle]
const excludedTitles = seeds.map(seed => seed.title).filter(title => !selectedTitles.includes(title))

await page.goto(`/operations/${operation.id}`)
await waitForAppReady(page)
const region = page.getByRole('region', { name: 'Operation', exact: true })
await expect(region.getByRole('listitem')).toHaveCount(20)
await expect(region.getByRole('heading', { name: longTitle })).toHaveCount(0)
const origin = new URL(page.url()).origin
const trigger = page.getByRole('button', { name: 'Export findings', exact: true })
const dialog = page.getByRole('dialog', { name: 'Export findings' })
const list = dialog.getByRole('list')
const selectAll = dialog.getByRole('checkbox', { name: 'Select all', exact: true })
const format = dialog.getByRole('combobox', { name: 'Format' })
const openWithEverySelected = async (): Promise<void> => {
await trigger.click()
await expect(list.getByRole('checkbox', { checked: true })).toHaveCount(seeds.length)
await expect(selectAll).toBeChecked()
// The sensitive-content warning wraps, and its bar must grow to hold it.
const warning = dialog.locator('.fui-MessageBar').filter({ hasText: 'Nothing is redacted automatically.' })
expect(await warning.evaluate(element => element.scrollHeight <= element.clientHeight)).toBeTruthy()
}
const selectSubset = async (): Promise<void> => {
await selectAll.click()
await expect(list.getByRole('checkbox', { checked: true })).toHaveCount(0)
for (const title of selectedTitles) {
await list.getByRole('checkbox', { name: title, exact: true }).check()
}
await expect(selectAll).toBeChecked({ indeterminate: true })
}
const download = async (
button: string, expectedFormat: FindingExportFormat,
): Promise<{ filename: string; content: Buffer }> => {
const exportRequest = page.waitForRequest(sent =>
sent.url().endsWith(`/api/operations/${operation.id}/findings/export`) && sent.method() === 'POST')
const downloaded = page.waitForEvent('download')
await dialog.getByRole('button', { name: button, exact: true }).click()
expect((await exportRequest).postDataJSON()).toEqual({
finding_ids: selectedTitles.map(findingId), format: expectedFormat, dashboard_base_url: `${origin}/`,
})
const file = await downloaded
await expect(dialog).toHaveCount(0)
await expect(trigger).toBeFocused()
return { filename: file.suggestedFilename(), content: readFileSync(await file.path()) }
}

await openWithEverySelected()
await expect(list.getByRole('checkbox').first()).toHaveAccessibleName('Export critical')
await expect(list.getByRole('checkbox').last()).toHaveAccessibleName(longTitle)
await selectAll.click()
await expect(list.getByRole('checkbox', { checked: true })).toHaveCount(0)
await expect(selectAll).not.toBeChecked()
await expect(dialog.getByRole('button', { name: 'Download Word', exact: true })).toBeDisabled()
await selectAll.click()
await expect(list.getByRole('checkbox', { checked: true })).toHaveCount(seeds.length)
await selectSubset()
await expect(format).toHaveValue('docx')
await expect(format.locator('option:checked')).toHaveText('Word (.docx)')
const word = await download('Download Word', 'docx')
// Named after the Operation and the browser's local time.
const filenameStart = `^copyrit-findings-${operation.name.toLowerCase().replace(/ /g, '-')}-\\d{4}-\\d{2}-\\d{2}-\\d{4}`
expect(word.filename).toMatch(new RegExp(`${filenameStart}\\.docx$`))
expect(word.content.subarray(0, 4).toString('latin1')).toBe('PK\x03\x04')
expect(word.content.includes('word/document.xml')).toBeTruthy()

// Reopening starts again from the whole Operation.
await openWithEverySelected()
await selectSubset()
await format.selectOption({ label: 'Markdown (.md)' })
await expect(list.getByRole('checkbox', { checked: true })).toHaveCount(selectedTitles.length)
const markdownFile = await download('Download Markdown', 'markdown')
expect(markdownFile.filename).toMatch(new RegExp(`${filenameStart}\\.md$`))
const markdown = markdownFile.content.toString('utf8')
expect(markdown.startsWith(`# Findings - ${operation.name}\n\n`)).toBeTruthy()
expect(markdown.match(/^## .*$/gm)).toEqual(selectedTitles.map(title => `## ${title}`))
for (const title of excludedTitles) {
expect(markdown).not.toContain(title)
}
expect(markdown).toContain(`**Description**\n\n\`\`\`\`text\n${description}\n\`\`\`\``)
const link = `${origin}/attacks/${attack.attack_result_id}/conversations/${attack.conversation_id}` +
`?findingEvidenceId=${evidence.item.id}`
// As in Word, the evidence bullet's objective is the link text.
expect(markdown.split('\n').filter(line => line.startsWith('- [') && line.endsWith(`](<${link}>)`))).toHaveLength(1)

await page.setViewportSize({ width: 390, height: 844 })
expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBeTruthy()
await openWithEverySelected()
const content = dialog.locator('.fui-DialogContent')
expect(await content.evaluate(element => element.scrollHeight > element.clientHeight
&& ['auto', 'scroll'].includes(getComputedStyle(element).overflowY))).toBeTruthy()
expect(await content.evaluate(element => element.scrollWidth <= element.clientWidth)).toBeTruthy()
expect(await dialog.evaluate(element => element.scrollWidth <= element.clientWidth)).toBeTruthy()
const longLabel = dialog.locator('label').filter({ hasText: longTitle })
expect(await longLabel.evaluate(element => element.scrollWidth <= element.clientWidth)).toBeTruthy()
const longBox = await longLabel.boundingBox()
const shortBox = await dialog.locator('label').filter({ hasText: /^Export low$/ }).boundingBox()
if (!longBox || !shortBox) throw new Error('Expected visible finding labels.')
expect(longBox.height).toBeGreaterThan(2 * shortBox.height)
expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBeTruthy()
await dialog.getByRole('button', { name: 'Cancel', exact: true }).click()
await expect(dialog).toHaveCount(0)
await expect(trigger).toBeFocused()

await page.goto(link)
await expect(page.getByText('Saved offline evidence.', { exact: true })).toBeVisible()
})
13 changes: 13 additions & 0 deletions frontend/src/components/Operations/FindingExportDialog.styles.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import { makeStyles, tokens } from '@fluentui/react-components'

import { mobileTouchTargetHeight } from '@/styles/touchTargets'

export const useFindingExportDialogStyles = makeStyles({
content: { display: 'flex', flexDirection: 'column', gap: tokens.spacingVerticalM },
messageBar: { flexShrink: 0 },
list: { margin: 0, padding: 0, listStyleType: 'none' },
row: { display: 'flex', flexWrap: 'wrap', alignItems: 'center', columnGap: tokens.spacingHorizontalS, minWidth: 0 },
checkbox: { flex: '1 1 12rem', minWidth: 0 },
label: { minWidth: 0, overflowWrap: 'anywhere' },
button: { ...mobileTouchTargetHeight },
})
Loading
Loading