Dropbox Tech Blog » Blog Archive » Improving Dropbox Performance: Retrieving Thumbnails

Dropbox Tech Blog » Blog Archive » Improving Dropbox Performance: Retrieving Thumbnails

For our web application, we use the Navigation Timing API to report back performance metrics. The API allows us to collect detailed metrics using JavaScript, for example DNS resolution time, SSL handshake time, page render time, and page load time: Instead of SPDY, we resorted to plain old HTTPS. We used a scheme where clients would send HTTP requests with multiple image urls (batch requests): GET https://photos.dropbox.com/thumbnails_batch?paths= /path/to/thumb0.jpg,/path/to/thumb1.jpg,[...],/path/to/thumbN.jpg The server sends back a batch response: HTTP/1.1 200 OK Cache-Control: public Content-Encoding: gzip Content-Type: text/plain Transfer-Encoding: chunked 1:data:image/jpeg;base64,4AAQ4BQY5FBAYmI4B[...] 0:data:image/jpeg;base64,I8FWC3EAj+4K846AF[...] 3:data:image/jpeg;base64,houN3VmI4BA3+BQA3[...] 2:data:image/jpeg;base64,MH3Gw15u56bHP67jF[...] [...] The response is: 1. Batched: we return all the images in a single plain-text response. Each image is on its own line, as a base-64-encoded data URI. Data URIs are required to make batching work with the web code rendering the photos page, since we can no longer just point an src tag to the response. JavaScript code sends the batch request with AJAX, splits the response and injects the data URIs directly into src tags. Base-64 encoding makes it easier to manipulate the response with JavaScript (e.g. splitting the lines). For mobile apps, we need to base64-decode the images before rendering them. 2. Progressive with chunked transfer encoding: on the backend, we fire off thumbnail requests in parallel to read the image data from our storage system. We stream the images back the moment they’re retrieved on the backend, without waiting for the entire response to be ready; this avoids head-of-line blocking, but also means we potentially send the images back out of order. We need to use chunked transfer encoding, since we don’t know the content length of the response ahead of time. We also need to prefix each line with the image index based on the order of request urls, to make sure the client can reorder the responses. On the client side, we can start interpreting the response the moment the first line is received. For web code we use progressive XMLHttpRequest; similarly for mobile apps, we simply read the response as it’s streamed down. 3. Compressed: we compress the response with gzip. Base64-encoding generally introduces 33% overhead. However, that overhead goes away after gzip compression. The response is no longer than sending the raw image data. 4. Cacheable: we mark the response as cacheable. When clients issue the same request in the future, we can avoid network traffic and serve the response out of cache. This does require us to make sure the batches are consistent however – any change in the request url would bypass the cache and require us to re-issue the network request.