Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 20 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ because it turns other people's test suites red.
coding as well as the bytes, so the time it says is left holds. The frames
between changes cost a few bytes each, so an hour at thirty
frames a second with a change every second fits in about a megabyte
(1 075 831 B at the smallest), and a `change_interval` as long as the film
or longer keeps one picture throughout (977 434 B for that hour). Set
(1 072 231 B at the smallest), and a `change_interval` as long as the film
or longer keeps one picture throughout (977 374 B for that hour). Set
`duration`, `change_interval`, `frame_rate` (whole rates from 1 to 60),
`keyframe_interval` - how far apart the frames a player can start from
are - `width`, `height` and `quality`. The picture is the gradient with the
Expand All @@ -55,6 +55,20 @@ because it turns other people's test suites red.
AV1 plays in current browsers, and an older player or a pipeline that
expects H.264 may refuse it - which is a test worth having. The window
lists it under Video.
- **Films: the `mp4` format.** The same films in an MP4 file: the same
settings, the same moving picture and the same manifest keys as `webm`, at
the exact size you ask for, and a WebM and an MP4 made from the same
settings carry the same AV1 frames. The index of the film comes first in
the file, so a browser starts playing before the whole file is in. To write
that index the film is coded twice, the second time from what the first
kept, which costs seconds - an hour of 1920x1080 changing every second is
coded in under one. The padding is free boxes at the end, so every size
from the smallest film up can be made, 5 GB and more included. A frame
between changes costs about 8 bytes, so an hour at thirty frames a second
with a change every second is 938 423 B at the smallest, and 867 623 B
with one picture throughout. The smallest film at the default settings is
3312 B. A pipeline that expects H.264 in an MP4 may refuse AV1, which is a
test worth having.
- **A length of time as a setting.** `duration`, `change_interval` and `keyframe_interval` take
`10s`, `1m30s`, `1h`, `500ms` or `59.9s`, to the millisecond. A bare number
is refused rather than read as seconds, and so is a length that does not
Expand Down Expand Up @@ -146,10 +160,10 @@ because it turns other people's test suites red.

### Changed

- **`empty-and-minimal` now includes a film.** The preset builds the smallest
file of every format this build has, so with `formats` left at `all` it
gains a `webm` target, and the recipe `tfg preset eject empty-and-minimal`
prints is a target longer. A manifest from a run of that preset records the
- **`empty-and-minimal` now includes the films.** The preset builds the
smallest file of every format this build has, so with `formats` left at
`all` it gains a `webm` and an `mp4` target, and the recipe
`tfg preset eject empty-and-minimal` prints is two targets longer. A manifest from a run of that preset records the
hash of that recipe, so its `recipe_hash` differs from one written before.
No generated file changes, and a run that names its `formats` is untouched.
- **An empty text box in the window reads "not set"** instead of "worked out
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

**Testing Files Generator** is a tool for QA engineers and developers who need real
files to test against - an upload form, a parser, anything that takes a file and
has an opinion about it. You pick one of its 27 formats and the size you want,
has an opinion about it. You pick one of its 28 formats and the size you want,
Comment thread
donislawdev marked this conversation as resolved.
and you get **exactly that**: ask for a 10 MB PDF and you get a PDF that a reader
will open, at 10 MB to the byte. Every run also leaves a manifest saying **what
your system should do with each file**, which is the part other generators leave
Expand All @@ -23,7 +23,7 @@ needs it finds out it exists.

- **Hit an exact size, to the byte** - ask for 10485761 bytes and get exactly
that, never a silently rounded file.
- **Write 27 real formats** - a generated PNG opens in an image viewer, a DOCX
- **Write 28 real formats** - a generated PNG opens in an image viewer, a DOCX
opens in Word, a ZIP extracts. Not padded zeros with an extension.
- **Say what should happen to each file** - the manifest carries an expected
outcome, so your test reads the assertion instead of you writing it out.
Expand Down Expand Up @@ -167,9 +167,9 @@ owns it, at the exact size you asked for:
| ⚙️ **Configuration** | `yaml`, `toml` |
| 🗜️ **Archives** | `zip`, `targz` (`.tar.gz`) |
| 🔊 **Audio** | `wav` |
| 🎞️ **Video** | `webm` (AV1, no sound) |
| 🎞️ **Video** | `webm`, `mp4` (both AV1, no sound) |

Coming next: `7z`, `mp3`, `mp4`.
Coming next: `7z`, `mp3`.

Most of them take settings of their own - image dimensions, JPEG quality, PDF
page count, rows and columns in a spreadsheet, what goes inside an archive. See
Expand Down Expand Up @@ -606,7 +606,7 @@ recipe. `tfg formats <id>` prints the allowed range or list for each:
| `avif`, `jpg`, `jxl` | `width`, `height`, `quality` |
| `ico` | `width`, `height`, `embed` |
| `wav` | `sample_rate`, `bit_depth`, `channels`, `content` |
| `webm` | `width`, `height`, `duration`, `change_interval`, `frame_rate`, `keyframe_interval`, `quality` |
| `webm`, `mp4` | `width`, `height`, `duration`, `change_interval`, `frame_rate`, `keyframe_interval`, `quality` |
| `zip` | `entries`, `entry_format`, `entry_size`, `compression`, `depth`, `directory_entries`, `password`, `encryption` |
| `targz` | `entries`, `entry_format`, `entry_size`, `compression`, `depth`, `directory_entries`, `entry_mode`, `entry_owner` |
| `docx` | `paragraphs` |
Expand Down
1 change: 1 addition & 0 deletions internal/format/all/all.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import (
_ "github.com/donislawdev/TestingFilesGenerator/internal/format/jxl"
_ "github.com/donislawdev/TestingFilesGenerator/internal/format/logfile"
_ "github.com/donislawdev/TestingFilesGenerator/internal/format/md"
_ "github.com/donislawdev/TestingFilesGenerator/internal/format/mp4"
_ "github.com/donislawdev/TestingFilesGenerator/internal/format/pdf"
_ "github.com/donislawdev/TestingFilesGenerator/internal/format/png"
_ "github.com/donislawdev/TestingFilesGenerator/internal/format/pptx"
Expand Down
80 changes: 3 additions & 77 deletions internal/format/avif/avif.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,28 +40,22 @@ package avif

import (
"context"
"encoding/binary"
"fmt"
"io"
"math/rand/v2"
"strconv"

"github.com/donislawdev/TestingFilesGenerator/internal/core"
"github.com/donislawdev/TestingFilesGenerator/internal/format"
"github.com/donislawdev/TestingFilesGenerator/internal/format/imagedim"
"github.com/donislawdev/TestingFilesGenerator/internal/format/isobmff"
)

const (
generatorVersion = "1"

// boxHeader is what every box costs before its content: a four byte
// length and a four character name.
boxHeader = 8

// maxFreePayload keeps a single box inside the four byte length field the
// container uses by default. Anything larger is spread over several boxes,
// which is measured to work rather than assumed.
maxFreePayload = 1<<31 - boxHeader
boxHeader = isobmff.BoxHeader

minDimension = 1
maxDimension = 16384
Expand Down Expand Up @@ -432,73 +426,5 @@ func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error {
if _, err := w.Write(coded); err != nil {
return err
}
return writePadding(ctx, w, m.seed, pad)
}

// writePadding fills the rest of the file with free boxes, without ever
// holding their content.
//
// Several boxes rather than one when the padding is larger than a box length
// can say. The step below keeps the leftover from landing between one and
// seven bytes, which no box could then carry.
func writePadding(ctx context.Context, w io.Writer, seed uint64, total int64) error {
rng := core.NewRand(seed)
buf := make([]byte, 32*1024)

for total > 0 {
payload := nextPayload(total)
if err := writeBoxHeader(w, payload); err != nil {
return err
}
if err := writeFiller(ctx, w, buf, rng, payload); err != nil {
return err
}
total -= boxHeader + payload
}
return nil
}

// nextPayload is how much filler the next free box carries.
//
// A box states its length in four bytes, so padding larger than that is spread
// over several boxes. The step down is what keeps the leftover from landing
// between one and seven bytes, which no box could then carry.
func nextPayload(total int64) int64 {
payload := total - boxHeader
if payload <= maxFreePayload {
return payload
}
payload = maxFreePayload
if total-boxHeader-payload < boxHeader {
payload -= boxHeader
}
return payload
}

func writeBoxHeader(w io.Writer, payload int64) error {
var b [boxHeader]byte
binary.BigEndian.PutUint32(b[:4], uint32(boxHeader+payload))
copy(b[4:], "free")
_, err := w.Write(b[:])
return err
}

func writeFiller(ctx context.Context, w io.Writer, buf []byte, rng *rand.Rand, size int64) error {
for left := size; left > 0; {
n := int64(len(buf))
if left < n {
n = left
}
select {
case <-ctx.Done():
return ctx.Err()
default:
}
core.FillRandomLE(buf[:n], rng)
if _, err := w.Write(buf[:n]); err != nil {
return err
}
left -= n
}
return nil
return isobmff.WritePadding(ctx, w, m.seed, pad)
}
94 changes: 94 additions & 0 deletions internal/format/isobmff/isobmff.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
// Package isobmff is what the formats built of ISO base media boxes share:
// the header every box starts with, and the free boxes their padding is
// carried in. AVIF and JPEG XL each held a copy of the same padding, and MP4
// would have been the third (docs/MP4-2026-10-07.md section 4).
package isobmff

import (
"context"
"encoding/binary"
"io"
"math/rand/v2"

"github.com/donislawdev/TestingFilesGenerator/internal/core"
)

// BoxHeader is what every box costs before its content: a four byte length
// and a four character name.
const BoxHeader = 8

// maxFreePayload keeps a single box inside the four byte length field the
// container uses by default. Anything larger is spread over several boxes,
// which is measured to work rather than assumed.
const maxFreePayload = 1<<31 - BoxHeader

// WriteBoxHeader writes the header of a box of kind whose content is payload
// bytes long, in the four byte length every box here uses.
func WriteBoxHeader(w io.Writer, kind string, payload int64) error {
var b [BoxHeader]byte
binary.BigEndian.PutUint32(b[:4], uint32(BoxHeader+payload))
copy(b[4:], kind)
_, err := w.Write(b[:])
return err
}

// WritePadding fills total bytes with free boxes, without ever holding their
// content - bytes drawn from seed, so the same file comes out every time.
// total is at least BoxHeader, because no box is shorter.
//
// Several boxes rather than one when the padding is larger than a box length
// can say. The step below keeps the leftover from landing between one and
// seven bytes, which no box could then carry.
func WritePadding(ctx context.Context, w io.Writer, seed uint64, total int64) error {
rng := core.NewRand(seed)
buf := make([]byte, 32*1024)

for total > 0 {
payload := nextPayload(total)
if err := WriteBoxHeader(w, "free", payload); err != nil {
return err
}
if err := writeFiller(ctx, w, buf, rng, payload); err != nil {
return err
}
total -= BoxHeader + payload
}
return nil
}

// nextPayload is how much filler the next free box carries.
//
// A box states its length in four bytes, so padding larger than that is spread
// over several boxes. The step down is what keeps the leftover from landing
// between one and seven bytes, which no box could then carry.
func nextPayload(total int64) int64 {
payload := total - BoxHeader
if payload <= maxFreePayload {
return payload
}
payload = maxFreePayload
if total-BoxHeader-payload < BoxHeader {
payload -= BoxHeader
}
return payload
}

func writeFiller(ctx context.Context, w io.Writer, buf []byte, rng *rand.Rand, size int64) error {
for left := size; left > 0; {
n := int64(len(buf))
if left < n {
n = left
}
select {
case <-ctx.Done():
return ctx.Err()
default:
}
core.FillRandomLE(buf[:n], rng)
if _, err := w.Write(buf[:n]); err != nil {
return err
}
left -= n
}
return nil
}
Loading
Loading