File Uploads Failed Because of 7 Bytes in a DNS Response

Hello. I'm Paul, a Software Engineer at DelightRoom.

I work in the DelightHub squad, improving the services we've acquired.
Curious about DelightHub? (Introduction to DelightHub)

In this post, I want to share a problem we experienced with Noji, a growing learning app at DelightHub, and how we solved it. Noji is a service for creating and reviewing flashcards, and it has a feature to import study materials created elsewhere as files. We started getting reports that this import process was failing, but only for some users.

The server logs showed successes, and trying it locally produced no issues either. As we traced the cause, we ended up digging past the application code and all the way down to DNS responses.

First, let's briefly go over some terms used in this post.

1. Server Returns 200, but the App Fails at DNS

I started by inspecting the file we received with the inquiry, but its compression structure and internal data were perfectly fine. Instead, we found this message in the app's error logs.

Unable to resolve host "our-services.s3.us-east-1.wasabisys.com":
No address associated with hostname

Noji uses Wasabi for file storage. This message meant the app couldn't find the IP for the Wasabi storage address. However, on the server side, the API issuing the upload address was happily returning a 200.

POST /api/…/upload_url → HTTP 200

The reason actually lies in our upload architecture. Noji doesn't have the API server receive large files and pass them to storage. The server only issues a signed upload URL, and the app sends the file directly to the storage. The server's 200 merely meant "successfully issued the URL." The failure was happening afterward, during the step where the app resolves the storage address via DNS. Since this phase bypasses the API server, there was no way it could leave a trace in our server logs.

So the question shifted from "Why did the upload API fail?" to "Why are some users failing to resolve the IP for the storage address?"

2. Works on LTE, but not on Wi-Fi

I manually queried the exact same address from my terminal.

dig @1.1.1.1 our-services.s3.us-east-1.wasabisys.com A

The result was normal. The list of IP addresses to connect to the storage came out just fine.

Meanwhile, we got another clue from a customer. They mentioned that the import process, which failed even after restarting the phone, worked as soon as they turned off Wi-Fi and switched to mobile data. Same phone, same app, same file, but changing the network changed the outcome. We needed to look at the network path where the DNS responses were traveling, rather than the phone itself.

3. A Response Exceeding 512 Bytes by 7 Bytes

This time, instead of using a public resolver, I queried the authoritative DNS of the Wasabi domain directly. The response contained one alias (CNAME) and 28 IP addresses, with a response size of 530 bytes when EDNS was enabled.

Without EDNS, the maximum DNS response size you can receive via UDP is 512 bytes. So I disabled EDNS and asked again using only UDP. +noedns is an option to disable EDNS, and +ignore is an option that shows the response exactly as received, preventing it from retrying via TCP even if truncated.

dig @ns0209.secondary.cloudflare.com \
  our-services.s3.us-east-1.wasabisys.com A \
  +noedns +ignore

flags: qr aa tc rd
ANSWER: 1
MSG SIZE rcvd: 71

A 71-byte response arrived with the truncation flag tc attached. The only answer left was a single CNAME, and not a single necessary IP address was there. When I sent the same question via TCP, I received a 519-byte answer containing all 28 IPs. It was a response exactly 7 bytes over the limit.

When a normal resolver receives a truncated response, it queries again via TCP to get the complete answer. My computer did exactly that. So for a user to experience a failure, there had to be a network path that receives the truncated response and then fails the subsequent TCP retry.

The phone itself was unlikely to be the cause, since the same phone succeeded on mobile data. The suspicious part was the DNS path on the Wi-Fi side. Some home routers' DNS relay features or certain resolvers strip out EDNS or drop TCP port 53 traffic entirely. Going through such a path leaves the phone with only a truncated response missing the IPs, resulting in "No address associated with hostname."

The logs from users who experienced the error perfectly matched this explanation. Device manufacturers, Android versions, and connection countries varied wildly, and there were even cases where the same user logged both successes and failures from the same public IP. Among the authoritative DNS servers, NS1-family servers returned only 24 IPs, coming in under the limit at 455 bytes. Rather than a bug with a specific device, it was an issue where outcomes diverged based on the path taken and the specific answer received. Of course, without the actual packets from the customers' networks, this was still just a hypothesis.

4. Same Storage, 23 Bytes Shorter Address

Looking at the Wasabi documentation, I noticed there was a shorter official address for the region we use: s3.wasabisys.com. The s3.us-east-1.wasabisys.com address we had been using was an alias. (Wasabi Region Service URLs)

When I asked under the same conditions, the response was 496 bytes. It fit neatly within the 512-byte limit despite containing the exact same 28 IPs, and there was no truncation flag.

If the contents were identical, why was there a 23-byte difference? DNS responses echo back the exact name queried. Because the subsequent CNAME and IP records refer to this name using a 2-byte pointer, the difference in response size directly equals the difference in the queried name's length. The long name is encoded as 41 bytes, and the short name as 18 bytes.

We started using the short official address, while also moving the bucket name from the hostname to the URL path. By combining these two changes, we reduced the name queried by DNS by 23 bytes.

If you do the math, the long address can fit up to 27 IPs into 512 bytes, while the short address can fit up to 29. The current count of 28 IPs landed perfectly right in the middle of that gap.

5. Reproducing the Failure

To confirm whether this difference actually led to failures, I whipped up a small program locally that only relays UDP (simulating a network where UDP works but TCP retries are blocked). It forwards the query from dig to the Wasabi authoritative DNS and returns the answer, but refuses TCP connections.

The commands I actually ran are the two lines below. I turned off EDNS with +noedns, and unlike before, I didn't include +ignore. This way, if the response is truncated, dig will query again via TCP. -p specifies the temporary port opened by the relay program.

dig @127.0.0.1 -p <중계 포트> our-services.s3.us-east-1.wasabisys.com A +noedns +time=2 +tries=1

dig @127.0.0.1 -p <중계 포트> s3.wasabisys.com A +noedns +time=2 +tries=1

The long address received a truncated response and failed with connection refused when trying to fallback to TCP (exit code 9). Leaving the relay program and authoritative DNS exactly the same and only switching to the shorter address, the 496-byte response arrived all at once, resulting in a success (exit code 0).

While it didn't perfectly replicate the customer's router, the conditions causing the failure and the method to sidestep them were now crystal clear.

6. How We Solved It

We had a few options on the table.

We went with the relatively straightforward last option. S3-compatible storage supports both putting the bucket name in the hostname (virtual-hosted-style) and in the URL path (path-style). Moving the bucket to the path shortens the name that DNS queries.

기존  <https://our-services.s3.us-east-1.wasabisys.com>
변경  <https://s3.wasabisys.com/our-services>

The storage, region, and file locations remain exactly the same. The only thing changing is the name DNS asks for. Since the app was already using whatever URL the server issued without modification, we just needed to change the part on the server that generates the upload URL. We spun up a separate client just for issuing upload URLs, to ensure no impact on other features like images or downloads that share the same storage.

def presigned_post_bucket
  @presigned_post_bucket ||= begin
    config = aws_s3_client.config
    endpoint = config.endpoint.dup
    endpoint.host = 's3.wasabisys.com' if endpoint.host == 's3.us-east-1.wasabisys.com'

    client = Aws::S3::Client.new(
      endpoint: endpoint.to_s,
      credentials: config.credentials,
      region: config.region,
      force_path_style: true,
      stub_responses: config.stub_responses
    )
    Aws::S3::Resource.new(client:).bucket(bucket.name)
  end
end

Thankfully, after the deployment, the same DNS errors vanished completely.

Wrapping Up

We want to work with folks at DelightRoom who enjoy tracking down the root cause to the bitter end, even when it lies outside the application layer. If you enjoy diving deep into problems, please apply to join us at DelightRoom.