Get best of the week

Gotta go fast. Optimizing IMAP email content requests

Hello everyone! In a previous article, I talked about how you can quickly synchronize the contents of a box in the local cache. Here I want to talk about the features of requesting the contents of letters and how best to request content without fear for a large consumption of traffic.

image

Let's quickly remember what we learned in the last article:

  • IMAP is a stateful protocol
  • To see the contents of the inbox, you must first select it with the SELECT command
  • To quickly synchronize the box we are in, you can use the NOOP command
  • In order not to sort messages from the local storage to update the mailbox we have already left, you can use CONDSTORE and QRESYNC, provided that your server supports the protocol extension data

Enough!


Let me remind you the command to request the letter body:

1 FETCH number (BODY[])

This will create a request to get the whole body of the letter and all attachments. Just see how long it takes to get a message in 42 paragraph of Lorem Ipsum and with a picture of 2 megabytes.

First, ask the size of the message on the server. This is done by the command:

1 FETCH 18871 (RFC822.SIZE)

RFC822.SIZE returns the size of the message in bytes:

* 18871 FETCH (RFC822.SIZE 3937793)

That is, as a result, our message takes up almost 4 megabytes.

Now, nevertheless, we will use the request for the full body of the letter and take a look at the time:

1 OK Fetch completed (0.007 + 3.265 secs).

3.3 seconds! And this is only one message with the attachment, and imagine them such will be the whole box. Then it will take more than a minute to download at least the twenty first ones.

You must admit that the business of a client who in 2020 cannot synchronize mail faster than in a minute is bad. But what to do?

Give me a bite once


If you rustle RFC3501 in clause 6.5.4 , which describes the possible parameters for the FETCH command, you will notice an interesting request:

BODY[<section>]<<partial>>

  • section - which part of the letter to get
  • partial - the size of this part

How is partial built? And it’s very easy. First, the byte from which you need to start reading is written through the point, and then how many bytes in general should be read:

BODY[<section>]<<0.1024>>

Here we request the part of the letter from zero byte to 1024.

Okay, what is section? First, I’ll talk about such a useful parameter in a FETCH query as BODYSTRUCTURE:

1 FETCH 18871 (BODYSTRUCTURE)

This parameter, as you probably understood from the signature, returns the structure of the letter in the form described in MIME-IMB .

* 18871 FETCH (BODYSTRUCTURE ((("text" "plain" ("charset" "utf-8") NIL NIL "quoted-printable" 25604 337 NIL NIL NIL NIL)("text" "html" ("charset" "utf-8") NIL NIL "quoted-printable" 29593 390 NIL NIL NIL NIL) "alternative" ("boundary" "--=_Part_763_774309787.1586268692") NIL NIL NIL)("image" "jpeg" ("name" "IMG_20200217_000236.jpg") NIL NIL "base64" 3880726 NIL ("attachment" ("filename" "IMG_20200217_000236.jpg")) NIL NIL) "mixed" ("boundary" "--=_Part_210_297656922.1586268692") NIL NIL NIL))


Just a look at this structure, and your head is spinning? Do not be afraid, now we’ll figure it out. Compare the opening and closing brackets.

(
BODYSTRUCTURE 
(
[1] (
[1.1] ("text" "plain" ("charset" "utf-8") NIL NIL "quoted-printable" 25604 337 NIL NIL NIL NIL)
[1.2] ("text" "html" ("charset" "utf-8") NIL NIL "quoted-printable" 29593 390 NIL NIL NIL NIL) "alternative" ("boundary" "--=_Part_763_774309787.1586268692") NIL NIL NIL
)
[2] ("image" "jpeg" ("name" "IMG_20200217_000236.jpg") NIL NIL "base64" 3880726 NIL ("attachment" ("filename" "IMG_20200217_000236.jpg")) NIL NIL) "mixed" ("boundary" "--=_Part_210_297656922.1586268692") NIL NIL NIL
)
)


You may notice that I put numbers near some brackets. This is section.
How to calculate them? The first bracket must be skipped, because it simply contains the answer to the request, then each opening bracket must be numbered according to the rule as the headings in the documents are numbered:

  • We number each opening bracket taking into account the previous section
  • If the section is nested, then the current one through the point is added to the previous number
  • If the section is not nested, increase its number by one


For example, in this case, in the first part that ends with “alternative” (that is, this is the part of the multipart / alternative letter, where we are free to choose which of the parts to display for the user), there are two sections that are numbered through a dot. I met a server where there may be three-level nesting (that is, [1.1.1], [1.1.2], etc).
Let's analyze the part [1.1] looking at the structure of all this stuff in the MIME-IMB document . Judging by it, the Content-Type header is first. It includes:

  • MIME type, here it is text / plain
  • Encoding (charset = utf8)


The following are two parameters that are written as NIL. Frankly, I did not understand what it was, but so far I have not needed it, so I will miss it. I apologize for such frivolity.
Next is the Content-Transfer-Encoding header, it is included in it, which describes the encoding mechanism, here it is quoted-printable. 
The following two numbers describe the size of the part in bytes and the number of lines, if possible. With their help, we can calculate how many bytes to take to display a certain number of lines.
The following lines that are not in this part:

  • Content-Id, which is used in the inline of the letter
  • Content-Description, a line that describes what this part is


For the other two parameters, I could not find a definite answer what it is, but one of these parameters may contain MD5 parts, which can sometimes be useful.
For part [2], everything is the same, except that it is an image, an attachment with a name, and base64 encoding. If it’s still not completely clear what is happening here, then on this site it’s perfectly laid out exactly how to calculate section.

What does it give? And the fact that at the stage of displaying the letter we can already request only the top part of the content and not load the attachments until the user himself enters the message and clicks the "download" button. All information for displaying attachments comes to us in BODYSCTRUCTURE so that the name, format and size can be displayed without loading the attachment itself. 

Let's move on to practice. We’ll ask for one kilobyte of message content without attachments, just to know what they sent us.

1 fetch 18871 (body[1.1]<0.1024>)
* 18871 FETCH (BODY[1.1]<0> {1024}
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus consecte=
tur enim in nisi venenatis, id varius tellus viverra. Praesent et enim te=
llus. Nunc vestibulum diam tortor, id posuere turpis tempor luctus. Vivam=
us molestie non nunc nec placerat. Cras finibus ut erat et tristique. Cur=
abitur vitae commodo risus. Etiam sed scelerisque erat. Quisque cursus bl=
andit finibus. Nullam ac lectus accumsan, molestie quam non, mollis urna.=
 Nulla at arcu in libero condimentum mollis ut non velit. Vestibulum sed =
risus et magna congue iaculis. Vestibulum nec interdum elit, ut commodo m=
auris. Nulla ipsum leo, vestibulum nec ligula non, elementum ullamcorper =
risus. Nunc et malesuada sem, id venenatis massa. Integer dolor ante, max=
imus in eleifend nec, ultricies ut risus. Mauris posuere eget tortor at p=
orttitor.=0AIn porta elementum ornare. Suspendisse aliquam, tortor sed al=
iquam bibendum, nulla ante rhoncus elit, placerat accumsan augue nibh non=
 est. Duis finibus vel tortor finibu)
1 OK Fetch completed (0.073 + 0.000 + 0.072 secs).


Some 100 milliseconds and we already see some of the content of the letter! This is just an excellent result, given that previously it took us almost 4 seconds to download the content of one letter. Then you can simply load the entire content of the letter in the background stream, from the outside it will seem that the letters are loaded instantly. All that was required was to look at the structure of the letter and download only what is required for a quick display. 
Only one moment. This request will make the message on the server appear as read. But you can fix this by adding only PEEK to the body request

1 fetch 18871 (BODY.PEEK[1.1]<0.1024>)
* 18871 FETCH (BODY[1.1]<0> {1024}
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus consecte=
tur enim in nisi venenatis, id varius tellus viverra. Praesent et enim te=
llus. Nunc vestibulum diam tortor, id posuere turpis tempor luctus. Vivam=
us molestie non nunc nec placerat. Cras finibus ut erat et tristique. Cur=
abitur vitae commodo risus. Etiam sed scelerisque erat. Quisque cursus bl=
andit finibus. Nullam ac lectus accumsan, molestie quam non, mollis urna.=
 Nulla at arcu in libero condimentum mollis ut non velit. Vestibulum sed =
risus et magna congue iaculis. Vestibulum nec interdum elit, ut commodo m=
auris. Nulla ipsum leo, vestibulum nec ligula non, elementum ullamcorper =
risus. Nunc et malesuada sem, id venenatis massa. Integer dolor ante, max=
imus in eleifend nec, ultricies ut risus. Mauris posuere eget tortor at p=
orttitor.=0AIn porta elementum ornare. Suspendisse aliquam, tortor sed al=
iquam bibendum, nulla ante rhoncus elit, placerat accumsan augue nibh non=
 est. Duis finibus vel tortor finibu)
1 OK Fetch completed (0.001 + 0.000 secs).


And voila! The letter remains unread and some of the content we received.
Everything becomes even easier if the PREVIEW request feature is implemented on your server.  

1 fetch 18871 (PREVIEW)
* 18871 FETCH (PREVIEW (FUZZY "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus consectetur enim in nisi venenatis, id varius tellus viverra. Praesent et enim tellus. Nunc vestibulum diam tortor, id posuere turpis t"))
1 OK Fetch completed (0.001 + 0.000 secs).


Here we don’t spend time on querying the structure at all and we get the message preview instantly. But do not forget that the query structure is useful for defining attachments so that they are not loaded into the idle.

Wait


Almost any mail client implements the “refresh” button if the user wants to receive new letters right now. But somehow this is not cool for our time, where there are notifications both in devices and in browsers. What does IMAP say about this? And he says IDLE . This operation holds the connection to the folder and notifies you of changes to the folder. Please note, not the mailbox, but the folders. To do this, you need the server to implement the IDLE feature. 

First, select the folder for which the server will send alerts, and then enable IDLE

1 SELECT Inbox
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded $MDNSent)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded $MDNSent \*)] Flags permitted.
* 18872 EXISTS
* 0 RECENT
* OK [UNSEEN 18685] First unseen.
* OK [UIDVALIDITY 1532079879] UIDs valid
* OK [UIDNEXT 20155] Predicted next UID
* OK [HIGHESTMODSEQ 26338] Highest
1 OK [READ-WRITE] Select completed (0.002 + 0.000 + 0.001 secs).
1 IDLE
+ idling


The answer "+ idling" notifies about the inclusion of the idle on the folder. What happens if a new letter arrives?

* 18873 EXISTS
* 1 RECENT


I sent myself the same letter, and Idle informed me that I should request letter 18873, that there were 18873 letters in the folder, and that one letter had just arrived.
Next, I will request this letter in another connection, we are interested in the letter with the answer EXISTS.

1 fetch 18873 (BODY.PEEK[1.1]<0.1024>)
* 18873 FETCH (BODY[1.1]<0> {1024}
---- Original Message ---- Tue, Apr 7, 2020, 17:11=0ASubject=
: Lorem Ipsum=0A  Lorem ipsum dolor sit amet, consectetur adipiscing elit=
. Vivamus consectetur enim in nisi venenatis, id varius tellus viverra. P=
raesent et enim tellus. Nunc vestibulum diam tortor, id posuere turpis te=
mpor luctus. Vivamus molestie non nunc nec placerat. Cras finibus ut erat=
…


It is very important to understand. IDLE requires a separate connection, so you cannot receive changes and request messages in the same session.
What else can IDLE do? He is able to notify about deleted letters and letters whose flags have changed. Let’s look at the letter for the sake of example, thereby putting the “/ seen” flag on it and delete the letter.

* 18873 FETCH (FLAGS (\Seen \Recent))
* OK Still here
* OK Still here
* 18873 EXPUNGE
* 18871 EXPUNGE
* 0 RECENT


I deleted the conversation (18873, 18871) and looked at another letter (FETCH response). Why did this letter become 18871? Because IMAP recounts the letter number if something has changed. Since it became the top one, its number also changed. 
With IDLE, we can quickly synchronize the state of the box, but it is unpleasant that it requires a separate connection. Could it be better? That is why I am here.

Yell how do you do


What if I tell you that there is a feature that allows you to receive notifications from the server in the same connection, and even specially configured for mailboxes, and more than one. It sounds like a fairy tale, but don’t go crazy, this is a real capability NOTIFY . He knows a lot, for example:

  • Configure specific folders from which we expect notifications
  • Listen to folder status changes (read letters, new letters)
  • Set the notification format, that is, what we want to see when changing the folder
  • Listen to folder name changes
  • Listen to folder metadata changes


Let's look at an example of how we can listen to folder status changes

1 notify set (inboxes (MessageNew FlagChange MessageExpunge))
1 OK NOTIFY completed (0.001 + 0.000 secs).


Now the server will send us notifications with folder statuses, for example, I will add a couple of messages to different folders

* STATUS INBOX/Ozon (MESSAGES 312 UIDNEXT 321 UNSEEN 48)
* STATUS "INBOX/Company News" (MESSAGES 178 UIDNEXT 179 UNSEEN 1)
* STATUS "INBOX/Company News" (MESSAGES 177 UIDNEXT 179 UNSEEN 0)


I will analyze the command:
First comes the NOTIFY SET command. Next, in brackets, we select which folders we will listen to:

  • Inboxes - for all folders that you can select
  • Personal - folders that are in the user's namespace
  • Subscribed - folders the user is subscribed to
  • Subtree - subtree of the folder to be specified
  • Mailboxes - here you can list the folders to listen to.
  • Selected - alert only for selected folders


And the parameters that are responsible for the alert filter:

  • MessageNew - if a new message arrives
  • FlagChange - if the flag has changed
  • MessageExpunge - if the message was deleted or moved


But with such a command, we cannot receive the parameters of a new, changed or deleted message. To do this, select the Selected parameter and specify what exactly to return. We can add another alert without deleting the previous one.

1 notify set status (selected (MessageNew (uid preview) MessageExpunge))


Here inside MessageNew we specify the parameters that the notification should return. I will choose Inbox and again I will throw to myself lorem ipsum.

* 18868 FETCH (UID 20157 PREVIEW (FUZZY "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus consectetur enim in nisi venenatis, id varius tellus viverra. Praesent et enim tellus. Nunc vestibulum diam tortor, id posuere turpis t"))


How do you like it? For the idle, we need to keep two connections, one of which also requests messages that the idle returned to us. Immediately they bring us everything on a silver platter. 
And so we can listen to folder name changes

1 notify set (inboxes (MailboxName))


Rename some folder and see the result.

* LIST () "/" 1111 ("OLDNAME" (aaaa))


And now we know that there was a folder “aaaa”, and became “1111”
Now you can listen to the change of flags and the removal of messages. To do this, use the FlagChange parameter

1 notify set (selected (MessageNew (uid) FlagChange MessageExpunge))


And when you change the message flags and delete, we get

* 18865 EXPUNGE
* 18864 FETCH (FLAGS ())
* 18864 FETCH (FLAGS (\Answered))


What's next


All these features help the mail client to work as quickly and conveniently as possible for the user. IDLE and NOTIFY notify the user of changes in folders, requesting part of the letter speeds up its loading. 
In the final article, I would like to talk about the search mechanism in IMAP and how it can be accelerated and reduce the load on the network. Thank you for reading to the end. 

More articles:


All Articles