Repository navigation
Support Elixir 1.16+, drop erlexec and tesla (v0.5.0) - #25
drozdzynski wants to merge 5 commits into
Conversation
- Require Elixir ~> 1.16 and bump version to 0.5.0 - Update all dependencies, drop coverex and mix_test_watch - Remove Bow.Exec and erlexec; add Bow.with_output/2 helper so versions can be generated with any tool (System.cmd, Vix, MuonTrap, ...) - Remove tesla; add Bow.Downloader behaviour with default :httpc based implementation streaming downloads to disk - Bow.Download no longer depends on Ecto - Use Config instead of Mix.Config, configurable test DB host/port - Add GitHub Actions CI with Postgres and S3 mock services
Tests write to tmp/, which does not exist in a fresh checkout.
48025c3 to
d4387f6
Compare
- Strip credentials and origin bound headers on cross-origin redirects (RFC 9110 section 15.4, same as Tesla FollowRedirects) - Run :httpc request in a Task so late messages never reach the caller - Make :timeout the total download time instead of idle time - Add :headers and :max_size options, remove temp file on download error - Document SSRF risk of user provided URLs and ExAws 2.7 hackney HEAD issue
| end | ||
| end | ||
|
|
||
| defp follow_redirect(downloader, url, path, opts, redirects_left, status, headers) do |
There was a problem hiding this comment.
IMO this should belong to client implementation. For example, I can use Tesla with Tesla.Middleware.FollowRedirects so it is encapsulated within client - Tesla handles it for me.
| tmp_path = Plug.Upload.random_file!("bow-output") | ||
| # random_file! does not support extensions, write output next to it and move it back | ||
| # so the file is still removed together with the process | ||
| output_path = tmp_path <> target.ext |
There was a problem hiding this comment.
Plug tracks the original tmp_path, not tmp_path <> target.ext, an exception after writing leaves the output behind. Some
try do
...
after
File.rm(output_path)
is required probably 🤔
| end | ||
|
|
||
| defp write_chunk(ref, file, size, chunk, state) do | ||
| IO.binwrite(file, chunk) |
There was a problem hiding this comment.
I think we should check {:error, reason} to minimize the chance file is corrupted
|
|
||
| case :httpc.request(:get, request, http_opts, | ||
| sync: false, | ||
| stream: :self, |
There was a problem hiding this comment.
I would think about using combination ofstream: {:self, :once} + stream_next() later. It sends one streaming event only after the consumer explicitly requests it so it is kind of backpressure mechanism. Currently if the network produces chunks faster than the task writes them, binaries accumulate in the task’s mailbox. That can increase memory usage.
Although I have never used it personally.
Wrap with_output/2 function call in try/after so output written before an exception does not stay in the tmp dir. Cancel the download when IO.binwrite fails instead of returning a truncated file.
Downloaders now follow redirects themselves and return the final URL, so clients like Req or Tesla can use their own redirect support. Bow.Downloader.Httpc gets the redirect loop, :max_redirects and cross-origin header filtering, with one deadline for all hops. Also use :file.write/2 for streamed chunks, IO.binwrite/2 raises instead of returning an error tuple.
Summary
Breaking release v0.5.0 modernising Bow and removing hard dependencies.
~> 1.16(tested on 1.16/OTP 26, 1.17/OTP 27, 1.19/OTP 28, 1.20/OTP 29), all dependencies updated,coverexandmix_test_watchremovedBow.Execanderlexecremoved. erlexec ≥ 2.0.8 requires OTP 27 and a native C++ port. Versions are generated intransform/3with any tool; newBow.with_output/2helper gives a temporary output path (with target extension, so ImageMagick/libvips pick the format), checks the file was written and sets it on the targetteslaremoved. NewBow.Downloaderbehaviour (config :bow, downloader: ...) with defaultBow.Downloader.Httpcbased on OTP:httpc: streams downloads to disk, verifies TLS certificates, redirects handled byBow.DownloadBow.Downloadno longer requires Ecto (Ecto.UUIDreplaced with:crypto)Mix.Config→Config,:warn→:warning, test DB host/port configurable viaTEST_DB_HOST/TEST_DB_PORTadobe/s3mock(docker) instead of the fakes3 Ruby gem andReqas ex_aws HTTP client (ex_aws 2.7 hackney adapter crashes onHEADresponses)See
CHANGELOG.mdfor the full list of breaking changes and migration notes.Migration
A custom HTTP client (proxy, different adapter, mocks in tests) is optional: implement
Bow.Downloaderand set it globally with
config :bow, downloader: MyApp.BowDownloaderor per call with thedownloader:option.Test plan
mix test --include ecto --include s3 --warnings-as-errorson Elixir 1.16/OTP 26, 1.17/OTP 27, 1.19/OTP 28, 1.20/OTP 29 (102 tests)mix compile --warnings-as-errors,mix format --check-formatted,mix deps.unlock --check-unusedBow.Downloader.Httpctested against a local Bandit server (streaming, redirects, 404, timeout, connection error) and the Req example from docs:tls_alert