diff --git a/.sampo/changesets/emails-attachments.md b/.sampo/changesets/emails-attachments.md new file mode 100644 index 00000000..d9ab8dcb --- /dev/null +++ b/.sampo/changesets/emails-attachments.md @@ -0,0 +1,5 @@ +--- +hex/posthog: minor +--- + +`PostHog.Emails.send/2` takes `cc`, `bcc` and `attachments` on a send with `template_key`. `cc` and `bcc` are one address or a list. Each attachment is a map with `filename`, `content` (the file's raw bytes) and optionally `content_type` and `content_id`. An email carries at most 10 files and 40 MB after Base64 encoding, about 29 MB of files. The SDK hashes each file, uploads the files PostHog doesn't have yet straight to PostHog's file storage, over https and without the secret key, and then sends the email with references to them. Plain-HTTP uploads are allowed only when `app_host` is plain HTTP, as in local development. If an upload fails, it sends nothing and returns `attachment_upload_failed`. An empty `reply_to`, `cc`, `bcc` or `attachments` list is now left out of the request. diff --git a/lib/posthog/emails.ex b/lib/posthog/emails.ex index 1bca407d..ea03b991 100644 --- a/lib/posthog/emails.ex +++ b/lib/posthog/emails.ex @@ -48,6 +48,20 @@ defmodule PostHog.Emails do idempotency_key: "password-reset:\#{reset.id}" ) + Copy finance and attach the receipt: + + PostHog.Emails.send( + to: "user@example.com", + cc: "finance@example.com", + template_key: "receipt", + subject: "Your receipt", + html: rendered_html, + attachments: [ + %{filename: "receipt.pdf", content: File.read!(pdf_path), content_type: "application/pdf"} + ], + distinct_id: "user_123" + ) + Read what happened to it: {:ok, %PostHog.Emails.Status{status: status}} = PostHog.Emails.get(message_id) @@ -71,13 +85,44 @@ defmodule PostHog.Emails do from `:api_host` with that key. When `:api_host` is a proxy that must not see the key, set `enable_local_evaluation: false` as well. + ## Attachments + + An email carries at most 10 files and 40 MB after Base64 encoding, about 29 MB of files. The + SDK checks both before any request of a send with files, counting each file and body as Base64 + and the subject as 75 bytes per 42 started, as PostHog does. Without `:text`, PostHog derives + the text from `:html` and counts it too, so the SDK counts the HTML twice. + + A send with files takes three steps: + + 1. The SDK hashes each file with SHA-256 and asks PostHog which files it doesn't have yet. + 2. It uploads each of those straight to PostHog's file storage with a plain `Req` request, + carrying only the headers PostHog gave. The upload doesn't go through + `:api_client_module`, so no secret key or header of your client reaches the storage. A + file PostHog already has, such as the same terms PDF sent to every customer, uploads + nothing again. + 3. Once every upload succeeded, it sends the email, which names each file by its hash. + + Uploads go only over https, so a file and its signed URL never travel unencrypted. When + PostHog names a plain-HTTP upload URL, the SDK uploads nothing, sends nothing, and returns + `api_error` on the attachment. An `:app_host` on plain HTTP, as in local development, allows + plain-HTTP uploads too. An upload never follows a redirect, which would carry the file to + another URL: the storage's redirect fails the upload with `attachment_upload_failed`. + + Your app must be able to reach the storage host that PostHog's answer names directly: an + upload doesn't go through a proxy, and it ignores your app's `Req.default_options`, which are + meant for other requests. If an upload fails, the SDK sends nothing and returns + `attachment_upload_failed` with what the storage answered. An upload gets the request timeout + plus ten seconds per MB of the file. It retries network errors and storage answers of HTTP 429, + 500, 502, 503 and 504, as a send retries, honouring `Retry-After`. + ## Timeouts, retries and idempotency Each request times out after `:email_request_timeout_ms` (15 seconds by default). A send retries network errors, rate limits, an idempotency key still in flight and an unavailable service `:email_request_max_retries` times (2 by default), waiting about 1 and then 2 seconds, or the server's `Retry-After` when it is longer. A `Retry-After` over 30 seconds fails at once - with `:retry_after_ms` set. A monthly quota is never retried. + with `:retry_after_ms` set. A monthly quota and the day's upload quota are never retried. + The upload steps of a send with files retry the same way. Every attempt sends the same idempotency key, so a retry never sends twice. Without `:idempotency_key`, the SDK makes a UUID for the call. When a job runner such as Oban @@ -100,28 +145,42 @@ defmodule PostHog.Emails do project. - `client_disabled`: the PostHog instance is not running or has no `:api_key`, so it sends nothing. - - `client_closed`: the PostHog instance stopped while a call waited to retry, so the retry - was not sent. + - `client_closed`: the PostHog instance stopped during a call, so the call made no further + request, and a send sent no email. + + A send with files can also return: + + - `too_many_attachments` or `message_too_large`: over 10 files, or over 40 MB after Base64 + encoding. The SDK checks both before any request, and PostHog checks them again. + - `attachment_upload_failed`: an upload to PostHog's file storage failed, so no email was + sent. The message has the storage's answer. + - `attachment_not_uploaded`: PostHog has no uploaded file with a hash the send names. + - `upload_quota_exceeded` (429): the project uploaded as many file bytes as it can today. + Files PostHog already has don't count. + - `field_not_accepted`: the project can't send files yet. An option the SDK refuses returns an error with an `invalid_*` code, such as `invalid_template_id`, and the name `:validation_error`, before any request. ## Testing - Requests go through the configured `:api_client_module`, so a `Mox` mock of - `PostHog.API.Client` sees every email request without network calls. + Requests to PostHog go through the configured `:api_client_module`, so a `Mox` mock of + `PostHog.API.Client` sees them without network calls. Attachment uploads go to the storage + host instead, so test a send with files against an answer that marks every file `present`. """ import Kernel, except: [send: 2] require Logger + alias PostHog.Emails.Attachments alias PostHog.Emails.Error alias PostHog.Emails.SendResult alias PostHog.Emails.Status alias PostHog.Emails.Warning @send_path "/api/projects/@current/messaging/send/email/" + @uploads_path "/api/projects/@current/messaging/attachments/uploads/" @status_path "/api/projects/@current/messaging/emails/" @us_app_host "https://us.posthog.com" @eu_app_host "https://eu.posthog.com" @@ -133,6 +192,8 @@ defmodule PostHog.Emails do "https://eu.posthog.com" => @eu_app_host } @max_retry_delay_ms 30_000 + # Enough of the storage's answer for an S3 error, whose XML is well under 1 KB. + @max_storage_answer_bytes 16_384 @log_metadata [posthog_skip_capture: true] @shared_fields [:to, :distinct_id, :idempotency_key, :max_retries] @@ -145,10 +206,14 @@ defmodule PostHog.Emails do :text, :from, :reply_to, + :cc, + :bcc, + :attachments, :sensitive, :tracking, :capture_engagement ] + @address_fields [:reply_to, :cc, :bcc] @typedoc """ Options for `send/2`. Pass `:template_id` or `:template_key`, never both. @@ -180,6 +245,15 @@ defmodule PostHog.Emails do - `:from` - a verified sender, such as `"Acme "`. Without it, the project's default sender. - `:reply_to` - one address or a list of addresses. + - `:cc` and `:bcc` - one address or a list of at most 10 bare addresses, such as + `"finance@example.com"`, that get a visible or a hidden copy. + - `:attachments` - a list of files: at most 10 files and 40 MB per email after Base64 + encoding, about 29 MB of files. Each is a map with `:filename`, `:content` (the file's + bytes, such as `File.read!/1` returns), and optionally `:content_type`, such as + `"application/pdf"`, and `:content_id`, which shows the file inline: reference it in + `:html` as `` with `content_id: "logo"`. Pass raw bytes, not Base64. + PostHog can't fetch a file from a path or URL. A file the SDK can't send returns the error + code `invalid_attachment`. See "Attachments" in `PostHog.Emails` for how files travel. - `:sensitive` - `true` when the email carries login links, reset tokens or codes. A sensitive email keeps its links unchanged unless `:tracking` turns clicks on, leaves its subject off engagement events, and keeps its content out of logs. Left out, the email takes the setting of @@ -198,7 +272,10 @@ defmodule PostHog.Emails do PostHog captures the email's engagement events. It still works, but a send can't set both. Sent only when you set it. - An option set to `nil` counts as left out. + An option set to `nil` counts as left out, and so does an empty `:reply_to`, `:cc`, `:bcc` or + `:attachments` list. A PostHog version that doesn't take a field yet, such as `:attachments`, + refuses the send with an error whose `:field` names it, and never drops the field to send the + rest. """ @type send_option() :: {:to, String.t()} @@ -213,10 +290,24 @@ defmodule PostHog.Emails do | {:text, String.t() | nil} | {:from, String.t() | nil} | {:reply_to, String.t() | [String.t()] | nil} + | {:cc, String.t() | [String.t()] | nil} + | {:bcc, String.t() | [String.t()] | nil} + | {:attachments, [attachment()] | nil} | {:sensitive, boolean() | nil} | {:tracking, tracking() | nil} | {:capture_engagement, boolean() | nil} + @typedoc """ + A file sent with the email. `:content` is the file's raw bytes, such as `File.read!/1` + returns. Don't pass content that is already Base64: the recipient would get the Base64 text. + """ + @type attachment() :: %{ + required(:filename) => String.t(), + required(:content) => binary(), + optional(:content_type) => String.t() | nil, + optional(:content_id) => String.t() | nil + } + @typedoc """ What PostHog tracks for an email sent with `:template_key`. See `:tracking` in `t:send_option/0`. @@ -257,6 +348,7 @@ defmodule PostHog.Emails do def send(name \\ PostHog, options) do with {:ok, prepared} <- prepare_send(options), {:ok, connection} <- connect(name, :send), + :ok <- upload(connection, prepared), {:ok, result} <- deliver(connection, prepared) do log_warnings(result) {:ok, result} @@ -308,11 +400,80 @@ defmodule PostHog.Emails do end) end + defp upload(_connection, %{files: []}), do: :ok + + defp upload(connection, prepared) do + max_retries = prepared.max_retries || connection.max_retries + + with {:ok, uploads} <- ask_where_to_upload(connection, prepared.files, max_retries), + :ok <- Attachments.check_https(uploads, plain_http?(connection)) do + put_files(connection, uploads, max_retries) + end + end + + # An app that talks to PostHog over plain HTTP, as in local development, may upload over it. + defp plain_http?(connection), do: URI.parse(connection.app_host).scheme == "http" + + defp put_files(_connection, [], _max_retries), do: :ok + + defp put_files(connection, [{file, upload} | uploads], max_retries) do + with :ok <- put_file(connection, file, upload, max_retries) do + put_files(connection, uploads, max_retries) + end + end + + defp ask_where_to_upload(connection, files, max_retries) do + headers = [{"content-type", "application/json"} | headers(connection)] + body = Jason.encode!(Attachments.upload_request(files)) + + with_retries(connection, max_retries, fn -> + connection + |> request(:post, @uploads_path, headers, body) + |> parse_uploads_reply(files) + end) + end + + # Ten seconds per MB: a file that fits still uploads over a link of about 1 Mbit/s. + defp upload_timeout_ms(connection, file), + do: connection.timeout_ms + div(file.size, 100) + + defp put_file(connection, file, upload, max_retries) do + timeout_ms = upload_timeout_ms(connection, file) + + connection + |> with_retries(max_retries, fn -> + case send_upload(connection, upload, file.content, timeout_ms) do + {:ok, status, _body, _retry_after_ms} when status in 200..299 -> + {:ok, :uploaded} + + {:ok, status, body, retry_after_ms} -> + {:error, + %{ + Attachments.upload_failed(file, upload.url, status, body) + | retry_after_ms: retry_after_ms + }} + + {:error, error} -> + {:error, error} + end + end) + |> case do + {:ok, :uploaded} -> + :ok + + {:error, %Error{code: "network_error"}} -> + {:error, Attachments.upload_failed(file, upload.url, nil, nil)} + + {:error, error} -> + {:error, error} + end + end + # Options defp prepare_send(options) when is_list(options) or is_map(options) do if is_map(options) or Keyword.keyword?(options) do - options |> Enum.reject(fn {_key, value} -> is_nil(value) end) |> Map.new() |> prepare() + options |> Enum.reject(&left_out?/1) |> Map.new() |> prepare() else invalid_options() end @@ -320,16 +481,23 @@ defmodule PostHog.Emails do defp prepare_send(_options), do: invalid_options() + # Options are often built generically, so an empty list counts as left out on both forms. + defp left_out?({key, []}) when key in [:attachments | @address_fields], do: true + defp left_out?({_key, value}), do: is_nil(value) + defp invalid_options, do: {:error, Error.new("invalid_form", "send takes a keyword list or a map of options.")} defp prepare(options) do with :ok <- check_recipient(options), {:ok, content} <- content(options), + {:ok, files} <- attachments(options), {:ok, idempotency_key} <- idempotency_key(options), {:ok, max_retries} <- max_retries(options), - {:ok, body} <- encode(content, options) do - {:ok, %{body: body, idempotency_key: idempotency_key, max_retries: max_retries}} + {:ok, body} <- + encode(put_if_any(content, :attachments, Enum.map(files, & &1.reference)), options) do + {:ok, + %{body: body, files: files, idempotency_key: idempotency_key, max_retries: max_retries}} end end @@ -397,10 +565,14 @@ defmodule PostHog.Emails do :sensitive ]) |> put_tracking(options) - |> put_reply_to(options)} + |> put_addresses(options)} end end + defp attachments(options) do + Attachments.prepare(Map.get(options, :attachments, []), options) + end + defp check_template_key(template_key) do if is_binary(template_key) and template_key =~ ~r/\A[a-zA-Z0-9_-]{1,400}\z/ do :ok @@ -440,11 +612,12 @@ defmodule PostHog.Emails do not_string?(options, :from) -> invalid("invalid_form", "from must be a string.", "from") - Map.has_key?(options, :reply_to) and not addresses?(options.reply_to) -> + field = + Enum.find(@address_fields, &(Map.has_key?(options, &1) and not addresses?(options[&1]))) -> invalid( "invalid_form", - "reply_to must be one address or a list of addresses.", - "reply_to" + "#{field} must be one address or a list of addresses.", + to_string(field) ) not is_boolean(Map.get(options, :capture_engagement, true)) -> @@ -468,6 +641,12 @@ defmodule PostHog.Emails do defp not_string?(options, field), do: Map.has_key?(options, field) and not is_binary(Map.fetch!(options, field)) + defp put_addresses(content, options) do + Enum.reduce(@address_fields, content, fn field, content -> + put_if_any(content, field, List.wrap(Map.get(options, field))) + end) + end + defp put_tracking(content, %{tracking: tracking}) when is_map(tracking), do: Map.put(content, :tracking, Map.reject(tracking, &is_nil(elem(&1, 1)))) @@ -481,11 +660,8 @@ defmodule PostHog.Emails do defp tracking?(_tracking), do: false - defp put_reply_to(content, %{reply_to: reply_to}) when is_binary(reply_to), - do: Map.put(content, :reply_to, [reply_to]) - - defp put_reply_to(content, %{reply_to: reply_to}), do: Map.put(content, :reply_to, reply_to) - defp put_reply_to(content, _options), do: content + defp put_if_any(content, _field, []), do: content + defp put_if_any(content, field, list), do: Map.put(content, field, list) defp addresses?(address) when is_binary(address), do: nonblank?(address) defp addresses?(addresses) when is_list(addresses), do: Enum.all?(addresses, &nonblank?/1) @@ -515,7 +691,7 @@ defmodule PostHog.Emails do {why, how} = case form do :template -> - {"A template made in PostHog sets its own sender, subject, content, tracking and sensitive setting, and follows the project's engagement setting.", + {"A template made in PostHog sets its own sender, copies, subject, content, attachments, tracking and sensitive setting, and follows the project's engagement setting.", "send content from your code with template_key"} :code -> @@ -618,7 +794,8 @@ defmodule PostHog.Emails do migrated_from: Map.get(config, :migrated_from), timeout_ms: config.email_request_timeout_ms, max_retries: config.email_request_max_retries, - sleep: Map.get(config, :email_sleep, &Process.sleep/1) + sleep: Map.get(config, :email_sleep, &Process.sleep/1), + upload_adapter: Map.get(config, :email_upload_adapter, :none) }} end end @@ -642,7 +819,7 @@ defmodule PostHog.Emails do {:error, Error.new( "client_closed", - "The PostHog instance #{inspect(name)} stopped before the retry, so #{nothing_done}." + "The PostHog instance #{inspect(name)} stopped during the call, so #{nothing_done}." )} end end @@ -702,6 +879,9 @@ defmodule PostHog.Emails do # Requests defp request(connection, method, path, headers, body) do + %{module: module, client: client} = connection.api_client + url = connection.app_host <> path + # The SDK parses the body itself, so a malformed acceptance stays an api_error with its status. options = [ @@ -713,23 +893,71 @@ defmodule PostHog.Emails do receive_timeout: connection.timeout_ms ] ++ if(body, do: [body: body], else: []) - %{module: module, client: client} = connection.api_client - - case within_deadline(connection.timeout_ms, fn -> - call_client(module, client, method, connection.app_host <> path, options) + case run_request(connection, connection.timeout_ms, fn -> + module.request(client, method, url, options) end) do - {:ok, %{status: status} = response} when is_integer(status) -> - {:ok, status, fields(Map.get(response, :body)), - retry_after_ms(Map.get(response, :headers))} + {:ok, status, body, retry_after_ms} -> {:ok, status, fields(body), retry_after_ms} + {:error, error} -> {:error, error} + end + end - {:error, _reason} -> - {:error, Error.new("network_error", "The request to PostHog failed.")} + # A bare request: not through the api_client, whose defaults, such as headers a custom client + # adds, are meant for PostHog, and without Req's steps and the app's Req.default_options, such as + # an auth or compress_body set for other requests. The storage gets the bytes and the headers + # PostHog signed the upload with, nothing else. Without Req's steps it never follows a redirect, + # so the file goes only to the URL PostHog signed. It reads only the start of the storage's + # answer, so a storage that answers without end can't fill the app's memory. + defp send_upload(connection, upload, content, timeout_ms) do + request = + Req.Request.new( + method: :put, + url: upload.url, + headers: for({name, value} <- upload.headers, do: {String.downcase(name), value}), + body: content, + options: [receive_timeout: timeout_ms] + ) + |> Map.put(:into, &keep_start/2) + |> with_upload_adapter(connection.upload_adapter) + + run_request(connection, timeout_ms, fn -> + case Req.Request.run_request(request) do + {_request, %Req.Response{} = response} -> {:ok, response} + {_request, exception} -> {:error, exception} + end + end) + end + + defp keep_start({:data, data}, {request, response}) do + body = response.body <> data + + if byte_size(body) < @max_storage_answer_bytes, + do: {:cont, {request, %{response | body: body}}}, + else: + {:halt, {request, %{response | body: binary_part(body, 0, @max_storage_answer_bytes)}}} + end + + # Req's own default adapter, which differs between Req versions, unless a test sets one. + defp with_upload_adapter(request, :none), do: request + + defp with_upload_adapter(request, {adapter, options}), + do: %{request | adapter: adapter, options: Map.merge(request.options, Map.new(options))} + + # Each request checks that the instance still runs, so a send stopped between its steps sends + # nothing more. + defp run_request(connection, timeout_ms, request) do + with :ok <- check_running(connection), + {:ok, %{status: status} = response} when is_integer(status) <- + within_deadline(timeout_ms, fn -> call_client(request) end) do + {:ok, status, Map.get(response, :body), retry_after_ms(Map.get(response, :headers))} + else + {:error, %Error{} = error} -> {:error, error} + {:error, _reason} -> {:error, Error.new("network_error", "The request to PostHog failed.")} end end # Finch raises, rather than returns, when no pooled connection frees up in time. - defp call_client(module, client, method, url, options) do - module.request(client, method, url, options) + defp call_client(request) do + request.() rescue exception in RuntimeError -> if String.starts_with?(exception.message, "Finch was unable to provide a connection"), @@ -829,22 +1057,22 @@ defmodule PostHog.Emails do delay_ms -> connection.sleep.(delay_ms) - retry(connection, max_retries, attempt_once, attempt + 1) + with_retries(connection, max_retries, attempt_once, attempt + 1) end end end - defp retry(connection, max_retries, attempt_once, attempt) do - with :ok <- check_running(connection) do - with_retries(connection, max_retries, attempt_once, attempt) - end - end - # A send carries an idempotency key, so a retry can't queue a second email. A status read changes nothing. + # An upload writes the same bytes to the same key, and S3 asks clients to retry its 500s. + defp retryable?(%Error{code: "attachment_upload_failed", status_code: status}), + do: status in [429, 500, 502, 503, 504] + defp retryable?(%Error{status_code: nil, code: code}), do: code == "network_error" defp retryable?(%Error{status_code: 409, code: code}), do: code == "idempotency_key_in_flight" # Waiting doesn't fix a monthly quota. - defp retryable?(%Error{status_code: 429, code: code}), do: code != "quota_limited" + defp retryable?(%Error{status_code: 429, code: code}), + do: code not in ["quota_limited", "upload_quota_exceeded"] + defp retryable?(%Error{status_code: status}), do: status == 503 defp retry_delay_ms(error, attempt, max_retries) do @@ -884,6 +1112,24 @@ defmodule PostHog.Emails do defp parse_send_reply(reply), do: reply_error(reply) + defp parse_uploads_reply({:ok, status, fields, _retry_after}, files) when status in 200..299 do + case Attachments.uploads(fields, files) do + {:ok, uploads} -> + {:ok, uploads} + + :error -> + {:error, + Error.new("api_error", "PostHog answered the upload request in a way the SDK can't use.", + status_code: status + )} + end + end + + defp parse_uploads_reply(reply, files) do + {:error, error} = reply_error(reply) + {:error, %{error | field: Attachments.field(error, files)}} + end + defp warnings(warnings) when is_list(warnings) do for %{} = warning <- warnings do %Warning{ diff --git a/lib/posthog/emails/attachments.ex b/lib/posthog/emails/attachments.ex new file mode 100644 index 00000000..6d1437e5 --- /dev/null +++ b/lib/posthog/emails/attachments.ex @@ -0,0 +1,349 @@ +defmodule PostHog.Emails.Attachments do + @moduledoc false + + alias PostHog.Emails.Error + + @max_files 10 + @max_message_bytes 40_000_000 + @headers_and_mime_boundaries_bytes 64 * 1024 + @subject_bytes_per_encoded_word 42 + @folded_encoded_word_bytes 75 + @limit "10 files and 40 MB per email after Base64 encoding, about 29 MB of files" + @fields [:filename, :content, :content_type, :content_id] + @field_names Enum.map(@fields, &Atom.to_string/1) + + @type file() :: %{ + field: String.t(), + filename: String.t(), + content: binary(), + sha256: String.t(), + size: pos_integer(), + reference: map() + } + + @type upload() :: %{url: String.t(), headers: [{String.t(), String.t()}]} + + @doc """ + Checks the files and the size of the email they make with its `:html`, `:text` and `:subject`, + and hashes each file. An email without files is left to PostHog to size, as nothing is + uploaded first. + """ + @spec prepare(term(), map()) :: {:ok, [file()]} | {:error, Error.t()} + def prepare(attachments, email) when is_list(attachments) do + # Hashing reads every byte, so it waits until the files are known to fit. + with :ok <- check_count(attachments), + {:ok, files} <- check_each(attachments), + :ok <- check_size(files, email) do + {:ok, Enum.map(files, &hash/1)} + end + end + + def prepare(_attachments, _email), + do: invalid("attachments must be a list of files.", "attachments") + + @doc "The body of the upload request: each distinct file once." + @spec upload_request([file()]) :: map() + def upload_request(files), + do: %{files: for(file <- distinct(files), do: %{sha256: file.sha256, size: file.size})} + + @doc """ + The files PostHog asks the SDK to upload, from its answer to the upload request, or `:error` + when the answer leaves a file out or names an upload the SDK can't make. + """ + @spec uploads(map(), [file()]) :: {:ok, [{file(), upload()}]} | :error + def uploads(%{"files" => answers}, files) when is_list(answers), + do: files |> distinct() |> uploads(answers, []) + + def uploads(_answer, _files), do: :error + + defp uploads([], _answers, uploads), do: {:ok, Enum.reverse(uploads)} + + defp uploads([file | files], answers, uploads) do + answer = Enum.find(answers, &match?(%{"sha256" => sha256} when sha256 == file.sha256, &1)) + + case upload(answer) do + :present -> uploads(files, answers, uploads) + {:ok, upload} -> uploads(files, answers, [{file, upload} | uploads]) + :error -> :error + end + end + + @doc """ + Refuses the uploads when one would go over plain HTTP, unless `plain_http?` allows it. The + error names the attachment and holds no URL. + """ + @spec check_https([{file(), upload()}], boolean()) :: :ok | {:error, Error.t()} + def check_https(_uploads, true), do: :ok + + def check_https(uploads, false) do + case Enum.find(uploads, fn {_file, upload} -> URI.parse(upload.url).scheme != "https" end) do + nil -> + :ok + + {file, _upload} -> + {:error, + Error.new( + "api_error", + "PostHog named an upload URL that isn't https, so #{file.filename} would travel unencrypted. No email was sent.", + field: file.field, + fix: + "Fix the file storage's URL in PostHog so it uses https. Plain-HTTP uploads are allowed only in local development, where :app_host is plain HTTP too." + )} + end + end + + @doc """ + Names the option a refusal of the upload request is about: `attachments`, or one of them. + PostHog names the upload request's `files`, and the caller passed `attachments`. A refusal of + the files without a field, such as a PostHog without the upload route, is about + `attachments`; one about the key, the scope, a rate limit or an outage names no option. + """ + @spec field(Error.t(), [file()]) :: String.t() | nil + def field(%Error{field: nil, status_code: status}, _files) when status in [400, 404, 413, 422], + do: "attachments" + + def field(%Error{field: "files"}, _files), do: "attachments" + def field(%Error{field: "files__" <> rest}, files), do: file_field(rest, files) + def field(%Error{field: field}, _files), do: field + + defp file_field(rest, files) do + with [index | _] <- String.split(rest, "__"), + {index, ""} <- Integer.parse(index), + %{field: field} <- Enum.at(distinct(files), index) do + field <> "__content" + else + _other -> "attachments" + end + end + + @doc "The error of an upload that failed, with what the storage answered, if anything." + @spec upload_failed(file(), String.t(), pos_integer() | nil, term()) :: Error.t() + def upload_failed(file, url, status, body) do + host = URI.parse(url).host + + answer = + if status, + do: "the storage answered #{s3_error(status, body)}", + else: "the request failed" + + Error.new( + "attachment_upload_failed", + "Uploading #{file.filename} to #{host} failed: #{answer}.", + status_code: status, + field: file.field, + why: + "The SDK sends an email only once each of its files is uploaded, so no email was sent.", + fix: + "Send again. If it keeps failing, check that your app can reach #{host} directly: uploads don't go through a proxy or your API client's settings." + ) + end + + defp check_count(attachments) when length(attachments) > @max_files do + {:error, + Error.new( + "too_many_attachments", + "An email carries at most #{@max_files} files, so no email was sent.", + field: "attachments", + why: "PostHog sends at most #{@limit}.", + fix: "Attach at most #{@max_files} files, and link to any others from the email body." + )} + end + + defp check_count(_attachments), do: :ok + + defp check_each(attachments) do + attachments + |> Enum.with_index() + |> Enum.reduce_while({:ok, []}, fn {attachment, index}, {:ok, files} -> + case file(attachment, "attachments__#{index}") do + {:ok, file} -> {:cont, {:ok, [file | files]}} + error -> {:halt, error} + end + end) + |> case do + {:ok, files} -> {:ok, Enum.reverse(files)} + error -> error + end + end + + defp check_size([], _email), do: :ok + + defp check_size(files, email) do + html = List.wrap(email[:html]) + derived_text? = email[:text] in [nil, ""] and html != [] + # Without text, PostHog derives it from the HTML and counts it too. The derived text drops + # tags and shortens entities and links, so it is never longer than the HTML. + bodies = if derived_text?, do: html ++ html, else: html ++ List.wrap(email[:text]) + sizes = Enum.map(files, & &1.size) ++ Enum.map(bodies, &byte_size/1) + subject = email[:subject] || "" + + if base64_message_bytes(sizes) + encoded_subject_bytes(subject) > @max_message_bytes do + {:error, + Error.new( + "message_too_large", + "These files make the email too large to send, so no email was sent.", + field: "attachments", + why: + "PostHog sends emails up to 40 MB. Base64 makes each file and body about a third larger, and the subject counts 75 bytes per 42 bytes started.", + fix: too_large_fix(derived_text?) + )} + else + :ok + end + end + + @too_large_fix "Keep the email under 40 MB after Base64 encoding, about 29 MB of files: attach fewer or smaller files, or link to large files from the email body." + + defp too_large_fix(false), do: @too_large_fix + + defp too_large_fix(true), + do: + @too_large_fix <> + " Without :text, PostHog derives the text from :html and counts it too, so pass a short :text to count only that." + + defp base64_message_bytes(sizes) do + Enum.reduce(sizes, @headers_and_mime_boundaries_bytes, fn size, total -> + total + ceil_div(ceil_div(size, 3) * 4 * 78, 76) + end) + end + + # A subject is sent as encoded words of 42 bytes each, folded onto lines of 75. + defp encoded_subject_bytes(subject), + do: ceil_div(byte_size(subject), @subject_bytes_per_encoded_word) * @folded_encoded_word_bytes + + defp ceil_div(dividend, divisor), do: div(dividend + divisor - 1, divisor) + + defp file(attachment, field) when is_map(attachment) and not is_struct(attachment) do + attachment = Map.reject(attachment, fn {_key, value} -> is_nil(value) end) + + with :ok <- check_keys(attachment, field), + :ok <- check_values(attachment, field) do + {:ok, + %{ + field: field, + filename: attachment.filename, + content: attachment.content, + size: byte_size(attachment.content), + reference: Map.take(attachment, [:filename, :content_type, :content_id]) + }} + end + end + + defp file(_attachment, field), + do: invalid("An attachment is a map with filename and content.", field) + + defp hash(file) do + sha256 = :crypto.hash(:sha256, file.content) |> Base.encode16(case: :lower) + %{file | reference: Map.put(file.reference, :sha256, sha256)} |> Map.put(:sha256, sha256) + end + + defp check_keys(attachment, field) do + unknown = attachment |> Map.keys() |> Enum.reject(&(&1 in @fields)) |> Enum.sort() + + cond do + Map.has_key?(attachment, :path) -> + invalid( + "PostHog can't fetch an attachment from a path or URL. Read the file, for example " <> + "with File.read!/1, and pass its bytes as content.", + field <> "__path" + ) + + unknown != [] -> + unknown_key(key_to_report(unknown), field) + + true -> + :ok + end + end + + defp check_values(attachment, field) do + cond do + not is_binary(attachment[:filename]) -> + invalid("An attachment needs its filename as a string.", field <> "__filename") + + String.trim(attachment.filename) == "" -> + invalid( + "An attachment's filename is empty, so the recipient couldn't tell what the file is.", + field <> "__filename" + ) + + not is_binary(attachment[:content]) -> + invalid( + "An attachment needs content: the file's bytes as a binary.", + field <> "__content" + ) + + attachment.content == "" -> + invalid( + "An attachment's content is empty, so the recipient would get nothing to open.", + field <> "__content" + ) + + key = Enum.find([:content_type, :content_id], ¬_string?(attachment, &1)) -> + invalid("An attachment's #{key} must be a string.", "#{field}__#{key}") + + true -> + :ok + end + end + + defp not_string?(attachment, key), + do: Map.has_key?(attachment, key) and not is_binary(attachment[key]) + + defp key_to_report(unknown_keys), + do: Enum.find(unknown_keys, hd(unknown_keys), &(&1 in @field_names)) + + defp unknown_key(key, field) when key in @field_names, + do: invalid(~s(Attachment keys are atoms: use :#{key}, not "#{key}".), "#{field}__#{key}") + + defp unknown_key(key, field) when is_atom(key) or is_binary(key), + do: invalid(keys_message(key), "#{field}__#{key}") + + defp unknown_key(key, field), do: invalid(keys_message(inspect(key)), field) + + defp keys_message(key), + do: "An attachment takes filename, content, content_type and content_id, not #{key}." + + defp invalid(message, field), + do: {:error, Error.new("invalid_attachment", message, field: field)} + + defp distinct(files), do: Enum.uniq_by(files, & &1.sha256) + + defp upload(%{"status" => "present"}), do: :present + + defp upload(%{"status" => "upload", "url" => url, "method" => "PUT", "headers" => headers}) + when is_binary(url) and is_map(headers) do + with %URI{scheme: scheme, host: host} + when scheme in ["https", "http"] and is_binary(host) and host != "" <- + URI.parse(url), + true <- Enum.all?(headers, fn {name, value} -> is_binary(name) and is_binary(value) end) do + {:ok, %{url: url, headers: Map.to_list(headers)}} + else + _unusable -> :error + end + end + + defp upload(_answer), do: :error + + defp s3_error(status, body) when is_binary(body) do + case {xml_text(body, "Code"), xml_text(body, "Message")} do + {nil, _message} -> + "HTTP #{status}" + + {code, nil} -> + "HTTP #{status} #{code}" + + {code, message} -> + "HTTP #{status} #{code}: #{message |> String.slice(0, 300) |> String.trim_trailing(".")}" + end + end + + defp s3_error(status, _body), do: "HTTP #{status}" + + defp xml_text(body, tag) do + case Regex.run(~r{<#{tag}>([^<]*)}, body, capture: :all_but_first) do + [text] when text != "" -> text + _none -> nil + end + end +end diff --git a/lib/posthog/emails/error.ex b/lib/posthog/emails/error.ex index 2aaed8d1..7ba1e00f 100644 --- a/lib/posthog/emails/error.ex +++ b/lib/posthog/emails/error.ex @@ -10,14 +10,19 @@ defmodule PostHog.Emails.Error do - `:name` - the category, one of `t:name/0`. - `:code` - PostHog's error code, such as `"quota_limited"` or `"recipient_suppressed"`, or an SDK code: `"missing_secret_key"`, `"missing_app_host"`, `"client_disabled"`, - `"client_closed"`, `"network_error"`, `"api_error"` (an answer without a code), `"unexpected_error"`, or an - `"invalid_*"` code for an option the SDK refused before any request. + `"client_closed"`, `"network_error"`, `"api_error"` (an answer without a code), + `"unexpected_error"`, `"too_many_attachments"` or `"message_too_large"` for files that + make the email too large, which the SDK checks before any request and PostHog checks again, + `"attachment_upload_failed"` when the storage refused an upload, or an `"invalid_*"` code + for an option the SDK refused before any request. - `:message` - human-readable message. - - `:status_code` - the HTTP status, when PostHog answered. + - `:status_code` - the HTTP status, when PostHog answered, or the storage's for + `"attachment_upload_failed"`. - `:why` and `:fix` - why PostHog refused the email and how to fix it, when it said. - `:docs_url` - a link to the docs for the error, when PostHog sent one. - `:field` - the option the error is about, such as `"to"`. - - `:retry_after_ms` - how long PostHog asked the client to wait before it sends again. + - `:retry_after_ms` - how long PostHog, or the storage, asked the client to wait before it + sends again. """ @typedoc "A coarse category of the failure." @@ -66,9 +71,15 @@ defmodule PostHog.Emails.Error do end defp name(_status, "missing_secret_key"), do: :missing_api_key + # The storage answered, so its status says nothing about PostHog's key or scope. + defp name(_status, "attachment_upload_failed"), do: :application_error defp name(401, _code), do: :missing_api_key defp name(nil, "network_error"), do: :service_unavailable defp name(nil, "invalid_" <> _), do: :validation_error + + defp name(nil, code) when code in ["too_many_attachments", "message_too_large"], + do: :validation_error + defp name(nil, _code), do: :application_error defp name(400, _code), do: :validation_error defp name(403, _code), do: :invalid_permission diff --git a/mix.exs b/mix.exs index 52cb4f6c..0004918e 100644 --- a/mix.exs +++ b/mix.exs @@ -82,6 +82,7 @@ defmodule PostHog.MixProject do # Development tools {:ex_doc, "~> 0.37", only: :dev, runtime: false}, {:logger_handler_kit, "~> 0.4", only: :test}, + {:bandit, "~> 1.7", only: :test}, {:credo, "~> 1.7", only: [:dev, :test], runtime: false}, {:mox, "~> 1.1", only: :test} ] diff --git a/test/posthog/emails_test.exs b/test/posthog/emails_test.exs index bdc7c8e9..b85d152b 100644 --- a/test/posthog/emails_test.exs +++ b/test/posthog/emails_test.exs @@ -15,6 +15,430 @@ defmodule PostHog.EmailsTest.ReqTestClient do defdelegate request(client, method, url, opts), to: Client end +defmodule PostHog.EmailsTest.HeaderedClient do + @moduledoc false + # The default client with Req.Test in place of the network, set up as an app might: its own + # authorization and an extra header on every request it makes. + @behaviour PostHog.API.Client + + alias PostHog.API.Client + + @impl true + def client(api_key, api_host) do + default = Client.client(api_key, api_host) + + client = + Req.merge(default.client, + plug: {Req.Test, PostHog.EmailsTest.ReqTestClient}, + headers: [{"authorization", "Bearer client-token"}, {"x-client", "app"}] + ) + + %{default | client: client} + end + + @impl true + defdelegate request(client, method, url, opts), to: Client +end + +defmodule PostHog.EmailsTest.OtherHost do + @moduledoc false + # A host the storage redirects to. It tells the test about every request it gets. + + def serve(conn, test) do + send(test, {:other_host, conn.method, conn.request_path}) + Plug.Conn.send_resp(conn, 200, "") + end +end + +defmodule PostHog.EmailsTest.FakePostHog do + @moduledoc false + # PostHog's email routes and its file storage as they behave, behind the SDK's real requests. + # The upload route signs an upload for each file PostHog doesn't have. The storage keeps a file + # only when its upload matches what was signed, as S3 checks a presigned PUT: the signed URL, + # the signed headers, no second credential or unsigned x-amz- header, and the bytes. The send + # route accepts an email only when PostHog has every file it names and the email fits PostHog's + # 40 MB estimate. A test reads back the stored files, the accepted emails and what the send route + # received, and can make a route's next request fail, wait or hold. + # + # The routes are :uploads and :send on PostHog, and :storage. Uploads are signed for + # https://files.example.com unless set_storage_url/2 names another host. + + @secret_key "phs_secret_key" + @uploads_path "/api/projects/@current/messaging/attachments/uploads/" + @send_path "/api/projects/@current/messaging/send/email/" + @storage_url "https://files.example.com" + @max_body 64_000_000 + @max_message_bytes 40_000_000 + @signature "X-Amz-Signature=signed-secret" + @signed_headers ["content-length", "x-amz-checksum-sha256"] + + @doc "Starts a fake for the calling test and routes the test's PostHog and storage requests to it." + def start! do + state = %{ + test: self(), + storage_url: @storage_url, + files: %{}, + signed: %{}, + messages: [], + received: %{uploads: [], storage: [], send: []}, + next: %{uploads: [], storage: [], send: []} + } + + fake = + ExUnit.Callbacks.start_supervised!(%{ + id: __MODULE__, + start: {Agent, :start_link, [fn -> state end]} + }) + + Req.Test.stub(PostHog.EmailsTest.ReqTestClient, &serve(&1, fake)) + Req.Test.stub(PostHog.EmailsTest.Storage, &serve(&1, fake)) + fake + end + + @doc "Stores files, as an earlier email that uploaded them would have." + def put_files(fake, files) do + Agent.update(fake, fn state -> + %{state | files: Map.merge(state.files, Map.new(files, &{sha256(&1), &1}))} + end) + end + + @doc "Deletes a stored file, as when it expires." + def delete_file(fake, bytes), + do: Agent.update(fake, &%{&1 | files: Map.delete(&1.files, sha256(bytes))}) + + @doc "Signs later uploads for the storage at `url`." + def set_storage_url(fake, url), do: Agent.update(fake, &%{&1 | storage_url: url}) + + @doc """ + Answers the route's next request with `reply` instead: `{status, json}` on PostHog's routes, + and `{status, s3_code, s3_message}` or a transport error's reason, such as `:econnrefused`, on + the storage. `{status, s3_code, s3_message, :endless}` follows the storage's error with + padding that doesn't end until the client stops reading, a little every 10 ms. + `{:redirect, status, location}` redirects the storage's next request to `location`. + """ + def fail_next(fake, route, reply), do: script(fake, route, {:reply, reply}) + + @doc "Makes the route take `ms` longer to answer its next request." + def delay_next(fake, route, ms), do: script(fake, route, {:delay, ms}) + + @doc """ + Holds the route's next request before it answers: the test gets `{:held, route, pid}`, and the + route answers once the test sends `:release` to `pid`. + """ + def hold_next(fake, route), do: script(fake, route, :hold) + + @doc "The stored files, by their hex SHA-256." + def files(fake), do: Agent.get(fake, & &1.files) + + @doc "The accepted emails as `{message_id, email}`, oldest first." + def messages(fake), do: Agent.get(fake, &Enum.reverse(&1.messages)) + + @doc """ + What the route received, oldest first: the JSON body on PostHog's routes, and the method, + path, query, headers and body on the storage. + """ + def received(fake, route), do: Agent.get(fake, &Enum.reverse(&1.received[route])) + + @doc "The answer PostHog gives for a file it doesn't have, signed for the default storage." + def upload_slot(bytes), do: upload_slot(@storage_url, sha256(bytes), byte_size(bytes)) + + defp upload_slot(storage_url, sha256, size) do + %{ + "sha256" => sha256, + "status" => "upload", + "url" => "#{storage_url}/team/1/#{sha256}?#{@signature}", + "method" => "PUT", + "headers" => %{ + "Content-Length" => to_string(size), + "x-amz-checksum-sha256" => checksum(sha256) + }, + "url_expires_at" => "2026-11-09T10:15:00Z" + } + end + + @doc "Serves PostHog and the storage, as a function plug for a real socket." + def serve(conn, fake) do + case conn.request_path do + @uploads_path -> posthog(conn, :uploads, fake) + @send_path -> posthog(conn, :send, fake) + "/api/" <> _other -> json(conn, 404, refusal("not_found", "Not found.", nil)) + _storage -> storage(conn, fake) + end + end + + defp script(fake, route, step), + do: Agent.update(fake, &update_in(&1.next[route], fn steps -> steps ++ [step] end)) + + # Records what the route received and takes its next scripted step, if any. + defp receive_request(fake, route, request) do + Agent.get_and_update(fake, fn state -> + {step, steps} = + case state.next[route] do + [step | steps] -> {step, steps} + [] -> {:answer, []} + end + + state = update_in(state.received[route], &[request | &1]) + {{step, state.test}, put_in(state.next[route], steps)} + end) + end + + defp run_step({:delay, ms}, _route, _test), do: Process.sleep(ms) + + defp run_step(:hold, route, test) do + send(test, {:held, route, self()}) + + receive do + :release -> :ok + end + end + + defp run_step(:answer, _route, _test), do: :ok + + # PostHog + + defp posthog(conn, route, fake) do + {:ok, body, conn} = Plug.Conn.read_body(conn, length: @max_body) + request = Jason.decode!(body) + + case receive_request(fake, route, request) do + {{:reply, {status, reply}}, _test} -> + json(conn, status, reply) + + {step, test} -> + run_step(step, route, test) + answer(conn, route, request, fake) + end + end + + defp answer(conn, route, request, fake) do + cond do + Plug.Conn.get_req_header(conn, "authorization") != ["Bearer #{@secret_key}"] -> + json(conn, 401, refusal("authentication_failed", "Invalid secret API key.", nil)) + + route == :uploads -> + sign_uploads(conn, request, fake) + + true -> + accept(conn, request, fake) + end + end + + defp sign_uploads(conn, %{"files" => files}, fake) when is_list(files) do + answers = + Agent.get_and_update( + fake, + &Enum.map_reduce(files, &1, fn file, state -> sign(file, state) end) + ) + + json(conn, 200, %{"files" => answers}) + end + + defp sign_uploads(conn, _request, _fake), + do: json(conn, 400, refusal("required", "This field is required.", "files")) + + defp sign(%{"sha256" => sha256, "size" => size}, state) do + if Map.has_key?(state.files, sha256) do + {%{"sha256" => sha256, "status" => "present", "expires_at" => "2026-12-16T09:00:00Z"}, + state} + else + slot = upload_slot(state.storage_url, sha256, size) + %URI{path: path, query: query} = URI.parse(slot["url"]) + {slot, put_in(state.signed[path], %{sha256: sha256, size: size, query: query})} + end + end + + defp accept(conn, email, fake) do + files = files(fake) + + missing = + email + |> Map.get("attachments", []) + |> Enum.find_index(&(not Map.has_key?(files, &1["sha256"]))) + + if missing do + json( + conn, + 400, + refusal( + "attachment_not_uploaded", + "A file this email names isn't uploaded, so PostHog didn't send the email.", + "attachments__#{missing}__sha256" + ) + ) + else + if message_bytes(email, files) > @max_message_bytes do + json(conn, 400, %{ + "type" => "invalid_request", + "code" => "message_too_large", + "detail" => "This email is too large to send, so PostHog didn't send it.", + "attr" => nil + }) + else + json(conn, 202, %{"message_id" => record_message(fake, email)}) + end + end + end + + # PostHog's estimate of the sent message: each file and body as folded Base64, 64 KiB for + # headers and MIME boundaries, and the subject as 75 bytes per 42 started. Without a text body, + # PostHog derives one from the HTML, here by dropping its tags, and counts it too. + defp message_bytes(email, files) do + html = email["html"] || "" + + text = + if email["text"] in [nil, ""], + do: String.replace(html, ~r/<[^>]*>/, ""), + else: email["text"] + + bodies = [byte_size(html), byte_size(text)] + sizes = for %{"sha256" => sha256} <- email["attachments"] || [], do: byte_size(files[sha256]) + subject = byte_size(email["subject"] || "") + + Enum.reduce(bodies ++ sizes, 64 * 1024 + div(subject + 41, 42) * 75, fn size, total -> + total + div(div(size + 2, 3) * 4 * 78 + 75, 76) + end) + end + + defp record_message(fake, email) do + Agent.get_and_update(fake, fn state -> + number = state.messages |> length() |> Kernel.+(1) |> Integer.to_string() + id = "01a18356-c56c-0001-e494-" <> String.pad_leading(number, 12, "0") + {id, %{state | messages: [{id, email} | state.messages]}} + end) + end + + defp refusal(code, detail, attr), + do: %{"type" => "validation_error", "code" => code, "detail" => detail, "attr" => attr} + + defp json(conn, status, body) do + conn + |> Plug.Conn.put_resp_content_type("application/json") + |> Plug.Conn.send_resp(status, Jason.encode!(body)) + end + + # Storage + + defp storage(conn, fake) do + {:ok, body, conn} = Plug.Conn.read_body(conn, length: @max_body) + headers = for {name, value} <- conn.req_headers, do: {String.downcase(name), value} + + request = %{ + method: conn.method, + path: conn.request_path, + query: conn.query_string, + headers: headers, + body: body + } + + case receive_request(fake, :storage, request) do + {{:reply, {:redirect, status, location}}, _test} -> + conn + |> Plug.Conn.put_resp_header("location", location) + |> Plug.Conn.send_resp(status, "") + + {{:reply, {status, code, message, :endless}}, _test} -> + endless_s3_error(conn, status, code, message) + + {{:reply, {status, code, message}}, _test} -> + s3_error(conn, status, code, message) + + {{:reply, reason}, _test} -> + Req.Test.transport_error(conn, reason) + + {step, test} -> + run_step(step, :storage, test) + store(conn, request, fake) + end + end + + defp store(conn, request, fake) do + signed = Agent.get(fake, & &1.signed[request.path]) + + cond do + request.method != "PUT" or signed == nil -> + s3_error(conn, 403, "AccessDenied", "Access Denied.") + + List.keymember?(request.headers, "authorization", 0) -> + s3_error(conn, 400, "InvalidArgument", "Only one auth mechanism allowed.") + + unsigned_amz_header?(request) -> + s3_error( + conn, + 403, + "AccessDenied", + "There were headers present in the request which were not signed." + ) + + not signature_matches?(request, signed) -> + s3_error(conn, 403, "SignatureDoesNotMatch", "The request signature does not match.") + + Base.encode64(:crypto.hash(:sha256, request.body)) != checksum(signed.sha256) -> + s3_error(conn, 400, "BadDigest", "The SHA256 you specified did not match.") + + true -> + Agent.update(fake, &put_in(&1.files[signed.sha256], request.body)) + + Plug.Conn.send_resp(conn, 200, "") + end + end + + # S3 ignores ordinary headers it wasn't asked to sign, but refuses an unsigned x-amz- one. + defp unsigned_amz_header?(request) do + Enum.any?(request.headers, fn {name, _value} -> + String.starts_with?(name, "x-amz-") and name not in @signed_headers + end) + end + + defp signature_matches?(request, signed) do + request.query == signed.query and + header(request, "x-amz-checksum-sha256") == checksum(signed.sha256) and + header(request, "content-length") == to_string(signed.size) + end + + defp header(request, name) do + case List.keyfind(request.headers, name, 0) do + {^name, value} -> value + nil -> nil + end + end + + defp s3_error(conn, status, code, message) do + conn + |> Plug.Conn.put_resp_content_type("application/xml") + |> Plug.Conn.send_resp(status, s3_error_xml(code, message)) + end + + defp endless_s3_error(conn, status, code, message) do + conn = + conn + |> Plug.Conn.put_resp_content_type("application/xml") + |> Plug.Conn.send_chunked(status) + + {:ok, conn} = Plug.Conn.chunk(conn, s3_error_xml(code, message)) + pad(conn, :binary.copy(" ", 65_536), 3_000) + end + + # Stops when the client closes the connection, or after 30 seconds. + defp pad(conn, _padding, 0), do: conn + + defp pad(conn, padding, chunks_left) do + Process.sleep(10) + + case Plug.Conn.chunk(conn, padding) do + {:ok, conn} -> pad(conn, padding, chunks_left - 1) + {:error, _closed} -> conn + end + end + + defp s3_error_xml(code, message), + do: + ~s(\n#{code}#{message}/team/1/file) + + defp sha256(bytes), do: :crypto.hash(:sha256, bytes) |> Base.encode16(case: :lower) + defp checksum(sha256), do: sha256 |> Base.decode16!(case: :lower) |> Base.encode64() +end + defmodule PostHog.EmailsTest do use ExUnit.Case, async: true @@ -24,6 +448,7 @@ defmodule PostHog.EmailsTest do alias PostHog.API alias PostHog.Emails alias PostHog.Emails.Error + alias PostHog.EmailsTest.FakePostHog setup :verify_on_exit! @@ -50,6 +475,10 @@ defmodule PostHog.EmailsTest do send(test_pid, {:slept, ms}) if context[:stop_instance_on_sleep], do: stop_supervised!(context.test) end) + |> Map.put( + :email_upload_adapter, + context[:upload_adapter] || {Req.Plug, plug: {Req.Test, PostHog.EmailsTest.Storage}} + ) start_link_supervised!({PostHog.Supervisor, config}) %{name: context.test} @@ -369,6 +798,804 @@ defmodule PostHog.EmailsTest do end end + # What a refused send leaves behind at PostHog: no email and no file. + defp nothing_sent?(fake), + do: FakePostHog.messages(fake) == [] and FakePostHog.files(fake) == %{} + + describe "send/2 with cc and bcc" do + @describetag config: [api_client_module: PostHog.EmailsTest.ReqTestClient] + @code_send [to: "user@example.com", template_key: "receipt", subject: "Hi", html: "

"] + + setup do + %{fake: FakePostHog.start!()} + end + + test "PostHog gets the copies, a string as a one-item list", %{name: name, fake: fake} do + assert {:ok, %Emails.SendResult{id: id}} = + Emails.send( + name, + @code_send ++ + [cc: "finance@example.com", bcc: ["audit@example.com", "archive@example.com"]] + ) + + assert [{^id, email}] = FakePostHog.messages(fake) + assert email["cc"] == ["finance@example.com"] + assert email["bcc"] == ["audit@example.com", "archive@example.com"] + end + + test "leaves out empty lists", %{name: name, fake: fake} do + assert {:ok, _} = + Emails.send( + name, + @code_send ++ [reply_to: [], cc: [], bcc: nil, attachments: []] + ) + + assert [{_id, email}] = FakePostHog.messages(fake) + refute Enum.any?(["reply_to", "cc", "bcc", "attachments"], &Map.has_key?(email, &1)) + end + + test "a template send refuses them and attachments, and sends nothing", %{ + name: name, + fake: fake + } do + for {field, value} <- [ + cc: "finance@example.com", + bcc: "audit@example.com", + attachments: [%{filename: "a.txt", content: "a"}] + ] do + assert {:error, %Error{code: "invalid_form", field: field_name} = error} = + Emails.send(name, [ + {field, value}, + to: "user@example.com", + template_id: @template_id + ]) + + assert field_name == to_string(field) + + assert error.why =~ + "sets its own sender, copies, subject, content, attachments, tracking and sensitive setting" + + assert error.fix =~ "template_key" + end + + assert nothing_sent?(fake) + end + + test "refuses cc and bcc that are not addresses", %{name: name, fake: fake} do + for {field, value} <- [cc: [:nope], bcc: " ", cc: %{a: 1}] do + assert {:error, %Error{code: "invalid_form", field: field_name}} = + Emails.send(name, @code_send ++ [{field, value}]) + + assert field_name == to_string(field) + end + + assert nothing_sent?(fake) + end + end + + describe "send/2 with attachments" do + @describetag config: [api_client_module: PostHog.EmailsTest.ReqTestClient] + @code_send [to: "user@example.com", template_key: "receipt", subject: "Hi", html: "

"] + @pdf <<37, 80, 68, 70, 45, 0, 255, 10>> + @logo "png bytes" + + setup do + %{fake: FakePostHog.start!()} + end + + defp sha256(bytes), do: :crypto.hash(:sha256, bytes) |> Base.encode16(case: :lower) + + defp with_files(attachments), do: @code_send ++ [attachments: attachments] + + test "uploads the files PostHog asks for, then sends their references", %{ + name: name, + fake: fake + } do + FakePostHog.put_files(fake, [@logo]) + + assert {:ok, %Emails.SendResult{id: id}} = + Emails.send( + name, + with_files([ + %{filename: "invoice.pdf", content: @pdf, content_type: "application/pdf"}, + %{filename: "logo.png", content: @logo, content_id: "logo"} + ]) + ) + + assert FakePostHog.files(fake) == %{sha256(@pdf) => @pdf, sha256(@logo) => @logo} + assert [{^id, email}] = FakePostHog.messages(fake) + + assert email["attachments"] == [ + %{ + "sha256" => sha256(@pdf), + "filename" => "invoice.pdf", + "content_type" => "application/pdf" + }, + %{"sha256" => sha256(@logo), "filename" => "logo.png", "content_id" => "logo"} + ] + end + + # The storage refuses a second credential, such as the client's authorization. + @tag config: [api_client_module: PostHog.EmailsTest.HeaderedClient] + test "uploads with only the headers PostHog signed, not the client's", %{ + name: name, + fake: fake + } do + assert {:ok, %Emails.SendResult{id: id}} = + Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + + assert FakePostHog.files(fake) == %{sha256(@pdf) => @pdf} + assert [{^id, %{"attachments" => [%{"sha256" => sha}]}}] = FakePostHog.messages(fake) + assert sha == sha256(@pdf) + end + + # The installed Req's default adapter, over a real socket, as in production. The local storage + # is plain HTTP, which an app on a plain-HTTP PostHog may upload to. + @tag upload_adapter: :none + @tag config: [ + api_client_module: PostHog.EmailsTest.ReqTestClient, + app_host: "http://localhost:8010" + ] + test "uploads over HTTP with Req's default adapter", %{name: name, fake: fake} do + {:ok, server} = + start_supervised({Bandit, plug: {&FakePostHog.serve/2, fake}, ip: :loopback, port: 0}) + + {:ok, {_ip, port}} = ThousandIsland.listener_info(server) + FakePostHog.set_storage_url(fake, "http://127.0.0.1:#{port}") + + assert {:ok, %Emails.SendResult{id: id}} = + Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + + assert FakePostHog.files(fake) == %{sha256(@pdf) => @pdf} + assert [{^id, _email}] = FakePostHog.messages(fake) + end + + # PostHog answers present with no upload URL, so the file can only be sent by reference. + test "sends a file PostHog already has by reference", %{name: name, fake: fake} do + FakePostHog.put_files(fake, [@pdf]) + + assert {:ok, _} = Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + + assert FakePostHog.files(fake) == %{sha256(@pdf) => @pdf} + assert [{_id, %{"attachments" => [%{"sha256" => sha}]}}] = FakePostHog.messages(fake) + assert sha == sha256(@pdf) + end + + test "sends a file attached twice as two references to one stored file", %{ + name: name, + fake: fake + } do + assert {:ok, _} = + Emails.send( + name, + with_files([ + %{filename: "a.pdf", content: @pdf}, + %{filename: "copy.pdf", content: @pdf} + ]) + ) + + assert FakePostHog.files(fake) == %{sha256(@pdf) => @pdf} + assert [{_id, email}] = FakePostHog.messages(fake) + + assert [ + %{"sha256" => sha, "filename" => "a.pdf"}, + %{"sha256" => sha, "filename" => "copy.pdf"} + ] = + email["attachments"] + + assert sha == sha256(@pdf) + end + + test "sends nothing after a failed upload, and returns the storage's answer", %{ + name: name, + fake: fake + } do + FakePostHog.put_files(fake, [@logo]) + + FakePostHog.fail_next( + fake, + :storage, + {400, "BadDigest", "The SHA256 you specified did not match the calculated checksum."} + ) + + assert {:error, %Error{} = error} = + Emails.send( + name, + with_files([ + %{filename: "logo.png", content: @logo}, + %{filename: "invoice.pdf", content: @pdf} + ]) + ) + + assert %Error{ + code: "attachment_upload_failed", + name: :application_error, + status_code: 400, + field: "attachments__1" + } = error + + assert error.message =~ "invoice.pdf" + assert error.message =~ "files.example.com" + assert error.message =~ "BadDigest" + assert error.message =~ "did not match the calculated checksum." + refute error.message =~ ".." + assert error.why =~ "no email was sent" + refute inspect(error) =~ "signed-secret" + refute inspect(error) =~ @pdf + + assert FakePostHog.files(fake) == %{sha256(@logo) => @logo} + assert FakePostHog.messages(fake) == [] + end + + @tag config: [ + api_client_module: PostHog.EmailsTest.ReqTestClient, + email_request_max_retries: 4 + ] + test "retries an upload the storage couldn't take, then sends", %{name: name, fake: fake} do + FakePostHog.fail_next(fake, :storage, {503, "SlowDown", "Please reduce your request rate."}) + FakePostHog.fail_next(fake, :storage, {429, "TooManyRequests", "Please slow down."}) + + FakePostHog.fail_next( + fake, + :storage, + {500, "InternalError", "We encountered an internal error."} + ) + + FakePostHog.fail_next(fake, :storage, :closed) + + assert {:ok, %Emails.SendResult{id: id}} = + Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + + assert FakePostHog.files(fake) == %{sha256(@pdf) => @pdf} + assert [{^id, _email}] = FakePostHog.messages(fake) + end + + @tag config: [ + api_client_module: PostHog.EmailsTest.ReqTestClient, + email_request_timeout_ms: 50 + ] + test "gives an upload ten seconds per MB on top of the request timeout", %{ + name: name, + fake: fake + } do + file = :binary.copy(<<1>>, 100_000) + # 100 KB at ten seconds per MB is 1 s on top of the 50 ms request timeout. + FakePostHog.delay_next(fake, :storage, 950) + + assert {:ok, _} = + Emails.send( + name, + with_files([%{filename: "a.bin", content: file}]) ++ [max_retries: 0] + ) + + assert FakePostHog.files(fake) == %{sha256(file) => file} + end + + @tag upload_adapter: :none + @tag config: [ + api_client_module: PostHog.EmailsTest.ReqTestClient, + app_host: "http://localhost:8010" + ] + test "doesn't follow the storage's redirect with the file", %{name: name, fake: fake} do + {:ok, storage} = + start_supervised({Bandit, plug: {&FakePostHog.serve/2, fake}, ip: :loopback, port: 0}) + + {:ok, other_host} = + start_supervised( + Supervisor.child_spec( + {Bandit, + plug: {&PostHog.EmailsTest.OtherHost.serve/2, self()}, ip: :loopback, port: 0}, + id: :other_host + ) + ) + + {:ok, {_ip, storage_port}} = ThousandIsland.listener_info(storage) + {:ok, {_ip, other_port}} = ThousandIsland.listener_info(other_host) + FakePostHog.set_storage_url(fake, "http://127.0.0.1:#{storage_port}") + elsewhere = "http://127.0.0.1:#{other_port}/elsewhere" + + for status <- [307, 308] do + FakePostHog.fail_next(fake, :storage, {:redirect, status, elsewhere}) + + assert {:error, %Error{code: "attachment_upload_failed", status_code: ^status} = error} = + Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + + assert error.field == "attachments__0" + assert error.message =~ "a.pdf" + refute inspect(error) =~ "elsewhere" + refute inspect(error) =~ "signed-secret" + refute_received {:other_host, _method, _path} + assert nothing_sent?(fake) + end + end + + # Reading the whole answer would outlast the upload's deadline of about a second. + @tag upload_adapter: :none + @tag config: [ + api_client_module: PostHog.EmailsTest.ReqTestClient, + app_host: "http://localhost:8010", + email_request_timeout_ms: 1_000 + ] + test "reads only the start of a storage answer that doesn't end", %{name: name, fake: fake} do + {:ok, server} = + start_supervised({Bandit, plug: {&FakePostHog.serve/2, fake}, ip: :loopback, port: 0}) + + {:ok, {_ip, port}} = ThousandIsland.listener_info(server) + FakePostHog.set_storage_url(fake, "http://127.0.0.1:#{port}") + + FakePostHog.fail_next( + fake, + :storage, + {403, "AccessDenied", "Request has expired.", :endless} + ) + + assert {:error, %Error{code: "attachment_upload_failed", status_code: 403} = error} = + Emails.send( + name, + with_files([%{filename: "a.pdf", content: @pdf}]) ++ [max_retries: 0] + ) + + assert error.message =~ "AccessDenied: Request has expired" + assert nothing_sent?(fake) + end + + # The refusal is the storage's next answer only, so a retry would succeed. + test "doesn't retry an upload the storage refused", %{name: name, fake: fake} do + FakePostHog.fail_next(fake, :storage, {403, "AccessDenied", "Request has expired."}) + + assert {:error, + %Error{ + code: "attachment_upload_failed", + name: :application_error, + status_code: 403 + }} = Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + + assert nothing_sent?(fake) + end + + test "a storage 401 doesn't blame the PostHog key", %{name: name, fake: fake} do + FakePostHog.fail_next(fake, :storage, {401, "Unauthorized", "Proxy needs a login."}) + + assert {:error, %Error{code: "attachment_upload_failed", name: :application_error}} = + Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + end + + test "an upload that keeps failing at the network sends nothing", %{name: name, fake: fake} do + FakePostHog.fail_next(fake, :storage, :econnrefused) + + assert {:error, %Error{code: "attachment_upload_failed", status_code: nil} = error} = + Emails.send( + name, + with_files([%{filename: "a.pdf", content: @pdf}]) ++ [max_retries: 0] + ) + + assert error.message =~ "files.example.com" + assert error.fix =~ "reach files.example.com directly" + assert error.field == "attachments__0" + refute inspect(error) =~ "signed-secret" + assert nothing_sent?(fake) + end + + test "a send whose instance stops between its steps sends nothing more", %{ + name: name, + fake: fake + } do + FakePostHog.hold_next(fake, :uploads) + + sending = + Task.async(fn -> Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) end) + + assert_receive {:held, :uploads, request} + stop_supervised!(name) + send(request, :release) + + assert {:error, %Error{code: "client_closed"} = error} = Task.await(sending) + assert error.message =~ "no email was sent" + assert nothing_sent?(fake) + end + + test "a template send takes empty lists as left out", %{name: name, fake: fake} do + assert {:ok, _} = + Emails.send(name, + to: "user@example.com", + template_id: @template_id, + reply_to: [], + cc: [], + bcc: [], + attachments: [] + ) + + assert [{_id, email}] = FakePostHog.messages(fake) + refute Enum.any?(["reply_to", "cc", "bcc", "attachments"], &Map.has_key?(email, &1)) + end + + test "retries an unavailable send with files, and the email goes out once", %{ + name: name, + fake: fake + } do + FakePostHog.fail_next(fake, :send, {503, %{"code" => "service_unavailable"}}) + + assert {:ok, %Emails.SendResult{id: id}} = + Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + + assert [{^id, _email}] = FakePostHog.messages(fake) + assert FakePostHog.files(fake) == %{sha256(@pdf) => @pdf} + end + + test "returns PostHog's refusal of the uploads, naming the attachment", %{ + name: name, + fake: fake + } do + FakePostHog.fail_next( + fake, + :uploads, + {400, + %{ + "type" => "invalid_request", + "code" => "message_too_large", + "detail" => + "These files are too large for one email, so PostHog didn't prepare the uploads.", + "attr" => "files__1__size", + "why" => "An email can be at most 40 MB after Base64 encoding, about 29 MB of files.", + "fix" => "Send fewer or smaller files." + }} + ) + + assert {:error, + %Error{ + code: "message_too_large", + status_code: 400, + field: "attachments__2__content", + why: "An email can be at most 40 MB" <> _ + }} = + Emails.send( + name, + with_files([ + %{filename: "a.png", content: @logo}, + %{filename: "b.png", content: @logo}, + %{filename: "c.pdf", content: @pdf} + ]) + ) + + FakePostHog.fail_next( + fake, + :uploads, + {400, %{"code" => "field_not_accepted", "attr" => "files", "detail" => "Not yet."}} + ) + + assert {:error, %Error{code: "field_not_accepted", field: "attachments"}} = + Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + + FakePostHog.fail_next( + fake, + :uploads, + {401, %{"code" => "authentication_failed", "detail" => "Bad key."}} + ) + + assert {:error, %Error{code: "authentication_failed", field: nil}} = + Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + + assert nothing_sent?(fake) + end + + test "a PostHog without the upload route sends nothing", %{name: name, fake: fake} do + FakePostHog.fail_next( + fake, + :uploads, + {404, %{"type" => "invalid_request", "code" => "not_found", "detail" => "Not found."}} + ) + + assert {:error, %Error{code: "not_found", status_code: 404, field: "attachments"}} = + Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + + assert nothing_sent?(fake) + end + + # Each refusal is the upload route's next answer only, so a retry succeeds. + test "doesn't retry the daily upload quota, and retries unavailable storage", %{ + name: name, + fake: fake + } do + FakePostHog.fail_next( + fake, + :uploads, + {429, %{"code" => "upload_quota_exceeded", "detail" => "Enough today."}} + ) + + assert {:error, %Error{code: "upload_quota_exceeded", status_code: 429, field: nil}} = + Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + + assert nothing_sent?(fake) + + FakePostHog.fail_next(fake, :uploads, {503, %{"code" => "delivery_unavailable"}}) + + assert {:ok, %Emails.SendResult{id: id}} = + Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + + assert [{^id, _email}] = FakePostHog.messages(fake) + assert FakePostHog.files(fake) == %{sha256(@pdf) => @pdf} + end + + test "an answer the SDK can't use sends nothing", %{name: name, fake: fake} do + slot = FakePostHog.upload_slot(@pdf) + + for answer <- [ + [], + [%{"sha256" => sha256(@logo), "status" => "present"}], + [%{"sha256" => sha256(@pdf), "status" => "copied"}], + [Map.delete(slot, "url")], + [Map.put(slot, "url", "ftp://files.example.com/a")], + [Map.put(slot, "method", "POST")], + [Map.put(slot, "headers", %{"Content-Length" => 8})] + ] do + FakePostHog.fail_next(fake, :uploads, {200, %{"files" => answer}}) + + assert {:error, %Error{code: "api_error", status_code: 200}} = + Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + end + + assert nothing_sent?(fake) + end + + test "refuses more than 10 files, and sends nothing", %{name: name, fake: fake} do + files = for i <- 1..11, do: %{filename: "#{i}.txt", content: "file #{i}"} + + assert {:error, + %Error{ + code: "too_many_attachments", + name: :validation_error, + field: "attachments" + } = error} = Emails.send(name, with_files(files)) + + assert error.fix =~ "10" + assert nothing_sent?(fake) + end + + test "refuses an email over 40 MB after Base64 encoding, and sends nothing", %{ + name: name, + fake: fake + } do + largest_that_fits = 29_182_821 + file = :binary.copy(<<0>>, largest_that_fits) + + for options <- [ + [html: "", attachments: [%{filename: "a", content: file <> <<0>>}]], + [html: "

", attachments: [%{filename: "a", content: file}]], + [html: "", text: "x", attachments: [%{filename: "a", content: file}]] + ] do + assert {:error, + %Error{code: "message_too_large", name: :validation_error, field: "attachments"} = + error} = Emails.send(name, @code_send ++ options) + + assert error.why =~ "40 MB" + assert error.fix =~ "29 MB" + end + + assert nothing_sent?(fake) + + FakePostHog.put_files(fake, [file]) + + assert {:ok, _} = + Emails.send( + name, + @code_send ++ [html: "", attachments: [%{filename: "a", content: file}]] + ) + + assert [{_id, _email}] = FakePostHog.messages(fake) + end + + # PostHog derives the text from the HTML and counts it too, never longer than the HTML. + test "counts the HTML twice without a text body", %{name: name, fake: fake} do + html = :binary.copy("a", 1_000_000) + files = [%{filename: "a.bin", content: :binary.copy(<<1>>, 28_000_000)}] + html_only = Keyword.put(with_files(files), :html, html) + + assert {:error, %Error{code: "message_too_large", field: "attachments"} = error} = + Emails.send(name, html_only) + + assert error.fix =~ ":text" + assert nothing_sent?(fake) + + with_text = html_only ++ [text: "Thanks"] + larger_files = [%{filename: "a.bin", content: :binary.copy(<<1>>, 29_500_000)}] + + assert {:error, %Error{code: "message_too_large"} = error} = + Emails.send(name, Keyword.put(with_text, :attachments, larger_files)) + + refute error.fix =~ ":text" + + assert {:ok, %Emails.SendResult{}} = Emails.send(name, with_text) + end + + test "uploads nothing to a plain-HTTP storage URL", %{name: name, fake: fake} do + FakePostHog.set_storage_url(fake, "http://files.example.com") + + assert {:error, %Error{code: "api_error", field: "attachments__0"} = error} = + Emails.send( + name, + with_files([ + %{filename: "a.pdf", content: @pdf}, + %{filename: "logo.png", content: @logo} + ]) + ) + + assert error.message =~ "a.pdf" + assert error.fix =~ "https" + refute inspect(error) =~ "files.example.com" + assert nothing_sent?(fake) + end + + @tag config: [ + api_client_module: PostHog.EmailsTest.ReqTestClient, + app_host: "http://localhost:8010" + ] + test "uploads over plain HTTP when the app talks to PostHog over plain HTTP", %{ + name: name, + fake: fake + } do + FakePostHog.set_storage_url(fake, "http://localhost:9000") + + assert {:ok, %Emails.SendResult{id: id}} = + Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) + + assert FakePostHog.files(fake) == %{sha256(@pdf) => @pdf} + assert [{^id, _email}] = FakePostHog.messages(fake) + end + + test "leaves the size of an email without files to PostHog", %{test: test, fake: fake} do + {:ok, server} = + start_supervised({Bandit, plug: {&FakePostHog.serve/2, fake}, ip: :loopback, port: 0}) + + {:ok, {_ip, port}} = ThousandIsland.listener_info(server) + host = "http://127.0.0.1:#{port}" + name = Module.concat(test, RealClient) + + config = + PostHog.Config.validate!( + api_host: host, + app_host: host, + api_key: "phc_project_key", + secret_key: "phs_secret_key", + supervisor_name: name, + enable_local_evaluation: false + ) + + start_link_supervised!(Supervisor.child_spec({PostHog.Supervisor, config}, id: name)) + html = :binary.copy("a", 40_000_000) + + assert {:error, %Error{code: "message_too_large", status_code: 400}} = + Emails.send(name, Keyword.put(@code_send, :html, html)) + + assert [%{"html" => ^html, "subject" => "Hi"}] = FakePostHog.received(fake, :send) + assert FakePostHog.messages(fake) == [] + end + + # With no subject and an empty body, these sizes leave 3, 73, 77, 147 and 151 bytes of the + # 40 MB estimate free, and 29_182_876 none: PostHog counts a subject as 75 bytes per 42 + # started. + for {label, subject, fits, refused} <- [ + {"of 42 ASCII bytes counts 75", String.duplicate("a", 42), 29_182_821, 29_182_824}, + {"of 43 bytes counts 150", String.duplicate("a", 43), 29_182_767, 29_182_770}, + {"of 21 two-byte characters counts 75", String.duplicate("é", 21), 29_182_821, + 29_182_824}, + {"that is empty counts nothing", "", 29_182_875, 29_182_876} + ] do + test "a subject #{label} toward the 40 MB estimate", %{name: name, fake: fake} do + send_file = fn size -> + Emails.send( + name, + Keyword.merge(@code_send, + subject: unquote(subject), + html: "", + attachments: [%{filename: "a", content: :binary.copy(<<0>>, size)}] + ) + ) + end + + assert {:error, %Error{code: "message_too_large"}} = send_file.(unquote(refused)) + assert nothing_sent?(fake) + + FakePostHog.put_files(fake, [:binary.copy(<<0>>, unquote(fits))]) + + assert {:ok, _} = send_file.(unquote(fits)) + assert [{_id, _email}] = FakePostHog.messages(fake) + end + end + + test "refuses an attachment it can't send, naming the field", %{name: name, fake: fake} do + for {attachments, field} <- [ + {%{filename: "a.txt", content: "a"}, "attachments"}, + {"invoice.pdf", "attachments"}, + {[[filename: "a.txt", content: "a"]], "attachments__0"}, + {[%{filename: "a.txt", content: "a"}, "b.txt"], "attachments__1"}, + {[%{filename: "a.txt", path: "/tmp/a.txt"}], "attachments__0__path"}, + {[%{filename: "a.txt", content: "a", cid: "logo"}], "attachments__0__cid"}, + {[%{"filename" => "a.txt", "content" => "a"}], "attachments__0__content"}, + {[%{filename: "a.txt"}], "attachments__0__content"}, + {[%{filename: "a.txt", content: ""}], "attachments__0__content"}, + {[%{filename: "a.txt", content: [97]}], "attachments__0__content"}, + {[%{filename: "a.txt", content: <<1::3>>}], "attachments__0__content"}, + {[%{content: "a"}], "attachments__0__filename"}, + {[%{filename: "", content: "a"}], "attachments__0__filename"}, + {[%{filename: " ", content: "a"}], "attachments__0__filename"}, + {[%{filename: :a, content: "a"}], "attachments__0__filename"}, + {[%{filename: "a.txt", content: "a", content_type: :pdf}], + "attachments__0__content_type"}, + {[%{filename: "a.txt", content: "a", content_id: 1}], "attachments__0__content_id"} + ] do + assert {:error, + %Error{code: "invalid_attachment", name: :validation_error, field: ^field}} = + Emails.send(name, with_files(attachments)) + end + + assert nothing_sent?(fake) + end + + test "asks for atom keys when an attachment has string keys", %{name: name, fake: fake} do + assert {:error, + %Error{code: "invalid_attachment", field: "attachments__0__content"} = error} = + Emails.send(name, with_files([%{"filename" => "a.txt", "content" => "a"}])) + + assert error.message =~ ~s(:content, not "content") + + assert {:error, %Error{field: "attachments__0__content"}} = + Emails.send( + name, + with_files([%{"filename" => "a.txt", "content" => "a", "cid" => "logo"}]) + ) + + assert nothing_sent?(fake) + end + + test "refuses an attachment key that has no string form", %{name: name, fake: fake} do + attachment = %{{:extra, :field} => true, filename: "a.txt", content: "a"} + + assert {:error, %Error{code: "invalid_attachment", field: "attachments__0"} = error} = + Emails.send(name, with_files([attachment])) + + assert error.message =~ "{:extra, :field}" + assert nothing_sent?(fake) + end + + test "a path points to reading the file", %{name: name} do + assert {:error, %Error{message: message}} = + Emails.send(name, with_files([%{filename: "a.pdf", path: "/srv/a.pdf"}])) + + assert message =~ "File.read" + end + + test "never repeats the file's content in an error", %{name: name} do + assert {:error, %Error{} = error} = + Emails.send(name, with_files([%{filename: 1, content: "SECRET-FILE-BYTES"}])) + + refute inspect(error) =~ "SECRET-FILE-BYTES" + end + + test "returns a send's refusal of a file that expired after its upload request", %{ + name: name, + fake: fake + } do + FakePostHog.put_files(fake, [@pdf]) + FakePostHog.hold_next(fake, :send) + + sending = + Task.async(fn -> Emails.send(name, with_files([%{filename: "a.pdf", content: @pdf}])) end) + + assert_receive {:held, :send, request} + FakePostHog.delete_file(fake, @pdf) + send(request, :release) + + assert {:error, + %Error{ + code: "attachment_not_uploaded", + field: "attachments__0__sha256", + status_code: 400 + }} = Task.await(sending) + + assert FakePostHog.messages(fake) == [] + end + end + describe "send/2 argument checks" do test "needs exactly one form", %{name: name} do assert {:error, %Error{code: "invalid_form"}} = Emails.send(name, to: "user@example.com") @@ -384,11 +1611,16 @@ defmodule PostHog.EmailsTest do end test "refuses an option neither form takes", %{name: name} do - assert {:error, %Error{code: "invalid_form", field: "cc", fix: "Remove cc and send again."}} = + assert {:error, + %Error{ + code: "invalid_form", + field: "headers", + fix: "Remove headers and send again." + }} = Emails.send(name, to: "user@example.com", template_id: @template_id, - cc: "x@example.com" + headers: %{"X-Campaign" => "welcome"} ) refute_requested() @@ -1041,3 +2273,69 @@ defmodule PostHog.EmailsTest do end end end + +defmodule PostHog.EmailsReqDefaultsTest do + # Req.default_options is app-wide, so this runs alone, after the async tests. + use ExUnit.Case, async: false + + alias PostHog.Emails + alias PostHog.EmailsTest.FakePostHog + + setup {Req.Test, :verify_on_exit!} + + setup context do + previous = Application.fetch_env(:req, :default_options) + + # Defaults an app might set for its own requests. The storage refuses either on an upload, as + # S3 does on a presigned PUT: a query that isn't the signed one, and gzipped bytes that no + # longer match the signed checksum. + Application.put_env(:req, :default_options, params: [app: "1"], compress_body: true) + + on_exit(fn -> + case previous do + {:ok, options} -> Application.put_env(:req, :default_options, options) + :error -> Application.delete_env(:req, :default_options) + end + end) + + config = + [ + api_host: "https://us.i.posthog.com", + api_key: "phc_project_key", + secret_key: "phs_secret_key", + # The local storage is plain HTTP, which an app on a plain-HTTP PostHog may upload to. + app_host: "http://localhost:8010", + api_client_module: PostHog.EmailsTest.ReqTestClient, + supervisor_name: context.test, + enable_local_evaluation: false + ] + |> PostHog.Config.validate!() + # The installed Req's default adapter over a real socket: Req.Test's plug would unzip a body. + |> Map.put(:email_upload_adapter, :none) + + start_link_supervised!({PostHog.Supervisor, config}) + %{name: context.test, fake: FakePostHog.start!()} + end + + test "an upload ignores the app's Req defaults", %{name: name, fake: fake} do + {:ok, server} = + start_supervised({Bandit, plug: {&FakePostHog.serve/2, fake}, ip: :loopback, port: 0}) + + {:ok, {_ip, port}} = ThousandIsland.listener_info(server) + FakePostHog.set_storage_url(fake, "http://127.0.0.1:#{port}") + file = "file bytes" + + assert {:ok, %Emails.SendResult{id: id}} = + Emails.send(name, + to: "user@example.com", + template_key: "receipt", + subject: "Hi", + html: "

", + attachments: [%{filename: "a.txt", content: file}] + ) + + sha256 = :crypto.hash(:sha256, file) |> Base.encode16(case: :lower) + assert FakePostHog.files(fake) == %{sha256 => file} + assert [{^id, %{"attachments" => [%{"sha256" => ^sha256}]}}] = FakePostHog.messages(fake) + end +end