Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
name: CI

on:
push:
branches: [master]
pull_request:

jobs:
test:
name: Elixir ${{ matrix.elixir }} / OTP ${{ matrix.otp }}
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix:
include:
# minimum supported
- elixir: "1.16"
otp: "26"
# latest
- elixir: "1.20"
otp: "28"
lint: true
env:
MIX_ENV: test
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: development
POSTGRES_HOST_AUTH_METHOD: trust
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 5s
--health-timeout 5s
--health-retries 10
s3:
image: adobe/s3mock
env:
COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS: test-bucket
ports:
- 4567:9090
steps:
- uses: actions/checkout@v5

- uses: erlef/setup-beam@v1
with:
elixir-version: ${{ matrix.elixir }}
otp-version: ${{ matrix.otp }}

- uses: actions/cache@v4
with:
path: |
deps
_build
key: ${{ runner.os }}-mix-${{ matrix.elixir }}-${{ matrix.otp }}-${{ hashFiles('mix.lock') }}
restore-keys: ${{ runner.os }}-mix-${{ matrix.elixir }}-${{ matrix.otp }}-

- run: mix deps.get

- if: matrix.lint
run: mix format --check-formatted

- if: matrix.lint
run: mix deps.unlock --check-unused

- if: matrix.lint
run: mix compile --warnings-as-errors

- run: mix ecto.create

- name: Wait for S3 mock
run: timeout 60 bash -c 'until curl -sf http://localhost:4567/ > /dev/null; do sleep 1; done'

- run: mix test --include ecto --include s3
2 changes: 2 additions & 0 deletions .tool-versions
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
erlang 28.5.0.2
elixir 1.19.5-otp-28
30 changes: 30 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,35 @@
# Changelog

# v0.5.0

Breaking changes:

* Minimum Elixir version is now 1.16
* `Bow.Exec` and `erlexec` dependency removed. Generate versions in `transform/3` with
`Bow.with_output/2` and any tool you like (`System.cmd/3`, Vix, MuonTrap, erlexec...).
`:exec_timeout` config removed
* `tesla` dependency removed. Remote files are downloaded with `:httpc` by default (`Bow.Downloader.Httpc`),
use `config :bow, downloader: MyDownloader` to plug in your own `Bow.Downloader` (e.g. Req or Tesla).
Downloaders follow redirects themselves and return the final URL used for the file name
* `Bow.Download.download(client, url)` is now `Bow.Download.download(url, opts)`
* `Bow.Ecto.cast_uploads/4` and `Bow.Ecto.download_params/3` take options (e.g. `downloader: ...`)
instead of a Tesla client
* `Bow.Download.download/2` errors are `{:error, %{status: status, headers: headers}}`,
`{:error, :too_many_redirects}`, `{:error, :max_size_exceeded}` and `{:error, :timeout}`
* Default downloader sends `user-agent: bow`, override it with the `:headers` option

Other changes:

* Updated dependencies (Ecto 3.10+, Plug 1.14+, ExAws 2.4+)
* Add `Bow.with_output/2` helper for writing processed files
* Downloaded files are streamed to disk instead of being loaded into memory
* `Bow.Download.download/2` and `Bow.Ecto.cast_uploads/4` options: `:headers`, `:max_size` (file size limit)
and `:timeout` (total download time)
* `Bow.Downloader.Httpc` removes credentials (`authorization`, `cookie`, ...) from `:headers` on redirect
to a different origin, `:max_redirects` option (defaults to 5)
* Document SSRF risk of downloading user provided URLs and the ExAws 2.7 hackney `HEAD` issue when loading files from `Bow.Storage.S3`
* fix: `Bow.Download` no longer requires Ecto

# v0.4.3

* fix: handle valid URL without file path
Expand Down
100 changes: 86 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,22 @@ File uploads for Elixir
```elixir
def deps do
[
{:bow, "~> 0.4.3"},
{:bow, "~> 0.5.0"},

# for AWS S3 support
{:ex_aws, "~> 2.0"},
{:ex_aws_s3, "~> 2.0"},

# for Bow.Exec
{:erlexec, "~> 1.7.0"}
{:ex_aws, "~> 2.4"},
{:ex_aws_s3, "~> 2.4"}
]
end
```

> **Note**
> Loading files with `Bow.Storage.S3` uses `ExAws.S3.download_file/4`, which sends a `HEAD` request.
> The hackney client of ExAws 2.7 crashes on `HEAD` responses, use the Req client instead:
> `config :ex_aws, http_client: ExAws.Request.Req` (requires `{:req, "~> 0.5"}`).

Bow requires Elixir 1.16 or newer.

## Usage

### Minimal uploader definition
Expand Down Expand Up @@ -65,8 +69,12 @@ defmodule AttachmentUploader do

# generate image thumbnail
def transform(source, target, :thumb) do
Bow.Exec.exec source, target,
"convert ${input} -strip -gravity Center -resize 250x175^ -extent 250x175 ${output}"
Bow.with_output(target, fn output_path ->
case System.cmd("convert", [source.path, "-resize", "250x175^", output_path]) do
{_, 0} -> :ok
{cmd_output, exit_code} -> {:error, exit_code: exit_code, output: cmd_output}
end
end)
end


Expand Down Expand Up @@ -186,6 +194,72 @@ defmodule AvatarUploader do
end
```

### Downloading remote files

`Bow.Ecto.cast_uploads/4` downloads files given as `remote_<field>_url` params
(e.g. `remote_avatar_url`). Files are streamed to disk with Erlang `:httpc` by default.

```elixir
changeset
|> Bow.Ecto.cast_uploads(params, [:avatar],
max_size: 10_000_000,
timeout: 15_000,
headers: [{"user-agent", "my-app"}]
)
```

The default downloader follows up to 5 redirects (`:max_redirects` option). On redirect to a different
origin, credentials like `authorization` and `cookie` are removed from `:headers`.

> **Warning**
> The URL comes from users, so it can point to your internal services, e.g. the
> `http://169.254.169.254/` cloud metadata endpoint (SSRF). Validate the URL before downloading
> it and always set `:max_size` to protect the disk.

You can use any HTTP client by implementing the `Bow.Downloader` behaviour
(see its docs for an example based on Req). Your downloader is then responsible for following
redirects without leaking credentials to a different origin:

```elixir
# config/config.exs
config :bow, downloader: MyApp.BowDownloader
```

### Processing files

`transform/3` can generate the version in any way you like. `Bow.with_output/2` helps with
the boring part: it gives you a temporary output path (with the target extension, so tools like
ImageMagick or libvips pick the right format), checks the file was written and sets it on the target.

With [Vix](https://hexdocs.pm/vix):

```elixir
def transform(source, target, :thumb) do
Bow.with_output(target, fn output_path ->
with {:ok, thumb} <- Vix.Vips.Operation.thumbnail(source.path, 250) do
Vix.Vips.Image.write_to_file(thumb, output_path)
end
end)
end
```

With an external program:

```elixir
def transform(source, target, :thumb) do
Bow.with_output(target, fn output_path ->
case System.cmd("convert", [source.path, "-resize", "250x175^", output_path]) do
{_, 0} -> :ok
{cmd_output, exit_code} -> {:error, exit_code: exit_code, output: cmd_output}
end
end)
end
```

`System.cmd/3` does not support timeouts. Version generation is limited by `:version_timeout`,
but the OS process may keep running after it. If you need to kill it, use a library like
[MuonTrap](https://hexdocs.pm/muontrap) or [erlexec](https://hexdocs.pm/erlexec).

### Using Bow in test environment

It is best to use local storage adapter when testing.
Expand All @@ -206,7 +280,8 @@ mix test
#### Testing ecto integration

```bash
# edit config/config.exs
# start postgres, or use your own and set TEST_DB_USERNAME, TEST_DB_PASSWORD, TEST_DB_HOST, TEST_DB_PORT
docker run --rm -d -p 5432:5432 -e POSTGRES_USER=development -e POSTGRES_HOST_AUTH_METHOD=trust postgres:16-alpine

# create test database
MIX_ENV=test mix ecto.create
Expand All @@ -218,11 +293,8 @@ mix test --only ecto
#### Testing S3 adapter

```bash
# install fake-s3 gem
gem install fakes3

# start fake-s3 server
fakes3 -r tmp/s3 -p 4567
# start S3 mock server
docker run --rm -d -p 4567:9090 -e COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=test-bucket adobe/s3mock

# run tests
mix test --only s3
Expand Down
10 changes: 6 additions & 4 deletions config/config.exs
Original file line number Diff line number Diff line change
@@ -1,22 +1,24 @@
use Mix.Config
import Config

if Mix.env() == :test do
config :bow, Bow.Repo,
username: System.get_env("TEST_DB_USERNAME") || "development",
password: System.get_env("TEST_DB_PASSWORD") || "",
database: "bow_ecto_test",
hostname: "localhost",
hostname: System.get_env("TEST_DB_HOST") || "localhost",
port: String.to_integer(System.get_env("TEST_DB_PORT") || "5432"),
pool: Ecto.Adapters.SQL.Sandbox

config :bow, ecto_repos: [Bow.Repo]

config :logger, level: :warn
config :logger, level: :warning
# fakes3 configuration
# https://github.com/jubos/fake-s3/wiki/Supported-Clients#elixir
config :ex_aws,
access_key_id: ["123", :instance_role],
secret_access_key: ["asdf", :instance_role],
region: "fakes3"
region: "fakes3",
http_client: ExAws.Request.Req

config :ex_aws, :s3,
scheme: "http://",
Expand Down
66 changes: 65 additions & 1 deletion lib/bow.ex
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ defmodule Bow do
storage_prefix: "priv/static/uploads", # storage directory prefix

store_timeout: 30_000, # single version upload timeout
exec_timeout: 15_000, # single command execution timeout
version_timeout: 60_000, # single version processing timeout
"""

def storage, do: Application.get_env(:bow, :storage, Bow.Storage.Local)
Expand Down Expand Up @@ -108,6 +108,70 @@ defmodule Bow do
end
end

@doc """
Run `fun` with an output path for `target` and set the result on it

Use it in `c:Bow.Uploader.transform/3` to write the processed file.
The output path has the same extension as `target` name, so tools choosing
the format based on extension (like libvips or ImageMagick) work out of the box.
The file is removed when the calling process exits.

`fun` must write the file and return `:ok` or `{:error, reason}`.

# thumbnail with Vix
def transform(source, target, :thumb) do
Bow.with_output(target, fn output_path ->
with {:ok, thumb} <- Vix.Vips.Operation.thumbnail(source.path, 250) do
Vix.Vips.Image.write_to_file(thumb, output_path)
end
end)
end

# thumbnail with ImageMagick
def transform(source, target, :thumb) do
Bow.with_output(target, fn output_path ->
case System.cmd("convert", [source.path, "-resize", "250x175^", output_path], stderr_to_stdout: true) do
{_, 0} -> :ok
{cmd_output, exit_code} -> {:error, exit_code: exit_code, output: cmd_output}
end
end)
end
"""
@spec with_output(t, (output_path :: Path.t() -> :ok | {:error, any})) ::
{:ok, t} | {:error, any}
def with_output(target, fun) when is_function(fun, 1) do
tmp_path = Plug.Upload.random_file!("bow-output")
# random_file! does not support extensions, write output next to it and move it back
# so the file is still removed together with the process
output_path = tmp_path <> target.ext
Comment on lines +143 to +146

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plug tracks the original tmp_path, not tmp_path <> target.ext, an exception after writing leaves the output behind. Some

try do
...
after
  File.rm(output_path)

is required probably 🤔


try do
case fun.(output_path) do
:ok ->
keep_output(target, output_path, tmp_path)

{:error, reason} ->
{:error, reason}

other ->
raise ArgumentError,
"expected :ok or {:error, reason} from with_output/2 function, got: #{inspect(other)}"
end
after
# no-op after a successful rename, removes partial output otherwise
File.rm(output_path)
end
end

defp keep_output(target, output_path, tmp_path) do
if File.exists?(output_path) do
File.rename!(output_path, tmp_path)
{:ok, set(target, :path, tmp_path)}
else
{:error, :output_not_found}
end
end

defp make(up, f0, fx, versions, opts) when is_list(versions) do
versions
|> Enum.map(&Task.async(fn -> make(up, f0, fx, &1, opts) end))
Expand Down
Loading
Loading