Monkeypatching the Ruby IMAP class to build a client for Exchange email servers not compliant with RFC standards

When trying to use Ruby Class Net::IMAP to automate some searching/arranging/deleting tasks for my office email I came across some Exchange responses that are not strictly compliant with the IMAP standard (currently 4rev1, described in RFC3501)

After loading the standard IMAP class

ruby-1.9.2-p290 :001 > require 'net/imap'
=> true

we can connect to the IMAP server by instantiating an Net::IMAP object that describes the type of connection used (IP address, TCP port and whether we should use SSL to encrypt the communication).
My company uses IMAP over SSL (port 993/tcp) with a 'PLAIN' authentication, so let's start with:

ruby-1.9.2-p290 :002 > imap= Net::IMAP.new( 'outlook.h3g.it', :port=>'imaps', :ssl=>true )
=> #<Net::IMAP:0x0000010109e118 @mon_owner=nil, @mon_count=0, @mon_mutex=#<Mutex:0x0000010109e0c8>, @host="outlook.h3g.it", @port="imaps", @tag_prefix="RUBY", @tagno=0, @parser=#<Net::IMAP::ResponseParser:0x0000010109e028 @str="* OK IMAP4 019 Ready\r\n", @pos=22, @lex_state=:EXPR_BEG, @token=nil, @flag_symbols={}>, @sock=#<OpenSSL::SSL::SSLSocket:0x0000010109d678>, @usessl=true, @responses={}, @tagged_responses={}, @response_handlers=[], @tagged_response_arrival=#<MonitorMixin::ConditionVariable:0x0000010109bb70 @monitor=#<Net::IMAP:0x0000010109e118 ...>, @cond=#<ConditionVariable:0x0000010109bb48 @waiters=[], @waiters_mutex=#<Mutex:0x0000010109baf8>>>, @continuation_request_arrival=#<MonitorMixin::ConditionVariable:0x0000010109bad0 @monitor=#<Net::IMAP:0x0000010109e118 ...>, @cond=#<ConditionVariable:0x0000010109baa8 @waiters=[], @waiters_mutex=#<Mutex:0x0000010109ba58>>>, @idle_done_cond=nil, @logout_command_tag=nil, @debug_output_bol=true, @exception=nil, @greeting=#<struct Net::IMAP::UntaggedResponse name="OK", data=#<struct Net::IMAP::ResponseText code=nil, text="IMAP4 019 Ready">, raw_data="* OK IMAP4 019 Ready\r\n">, @client_thread=#<Thread:0x00000100892c80 run>, @receiver_thread=#<Thread:0x0000010109afb8 run>>

So far so good, but when I introduce myself to the server I receive an error:

ruby-1.9.2-p290 :003 > imap.authenticate( 'PLAIN', my_name, my_password )
Net::IMAP::ResponseParseError: unexpected token CRLF (expected SPACE)
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:3235:in `parse_error'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:3087:in `match'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:2058:in `continue_req'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:2045:in `response'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1973:in `parse'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1124:in `get_response'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1036:in `receive_responses'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1023:in `block in initialize'

It seems that our client is expecting a T_SPACE while server is sending T_CRLF. Interesting to say, Microsoft states that Exchange is "compatible with RFC3501" but also that "(AUTH=PLAIN not supported)" (http://technet.microsoft.com/en-us/library/ff848256.aspx). Why?? It seems to me that once again they are not able to understand a clear (A)BNF specification on a standard RFC:

RFC3501: INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1
9. Formal Syntax
response = *(continue-req / response-data) response-done
continue-req = "+" SP (resp-text / base64) CRLF
response-data = "*" SP (resp-cond-state / resp-cond-bye / mailbox-data / message-data / capability-data) CRLF
response-done = response-tagged / response-fatal
response-fatal = "*" SP resp-cond-bye CRLF ; Server closes connection immediately
response-tagged = tag SP resp-cond-state CRLF

RFC clearly says that a SP(ace) is needed after the "+" sign, as correctly defined in the 'continue_req' method:

require 'net/imap'
module Net

class IMAP

class ResponseParser

def continue_req

match(T_PLUS)
match(T_SPACE)
return ContinuationRequest.new(resp_text, @str)

end

end

end

end

however we can try to fix (but I should say 'break') it to suit the Exchange server's needs, deleting the 'match(T_SPACE)' statement:

require 'net/imap'
module Net

class IMAP

class ResponseParser

def continue_req

match(T_PLUS)
return ContinuationRequest.new(resp_text, @str)

end #def continue_req

end #class ResponseParser

end #class IMAP

end #module Net

And then try again:

ruby-1.9.2-p290 :004 > imap.authenticate( 'PLAIN', my_name, my_password )
=> #, raw_data="RUBY0001 OK AUTHENTICATE completed.\r\n">

And now it's ok.
But testing some other IMAP command we soon face another error:

ruby-1.9.2-p290 :005 > imap.noop
=> #, raw_data="RUBY0002 OK NOOP completed.\r\n">

ruby-1.9.2-p290 :006 > imap.status( 'INBOX', 'MESSAGES' )
Net::IMAP::ResponseParseError: unexpected token SPACE (expected CRLF)
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:3235:in `parse_error'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:3087:in `match'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:2051:in `response'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1973:in `parse'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1124:in `get_response'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1036:in `receive_responses'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/imap.rb:1023:in `block in initialize'

Exchange server (sometimes?) adds an extra space before CRLF (again violating the RFC command syntax listed above).
This time we need to change the response method:

require 'net/imap'
module Net

class IMAP

class ResponseParser

def response

token = lookahead
case token.symbol
when T_PLUS

result = continue_req

when T_STAR

result = response_untagged

else

result = response_tagged

end
match(T_CRLF)
match(T_EOF)
return result

end

end

end

end

adding a 'match(T_SPACE) if lookahead.symbol == T_SPACE' just before the CRLF match:

require 'net/imap'
module Net

class IMAP

class ResponseParser

def response

token = lookahead
case token.symbol
when T_PLUS

result = continue_req

when T_STAR

result = response_untagged

else

result = response_tagged

end
match(T_SPACE) if lookahead.symbol == T_SPACE
match(T_CRLF)
match(T_EOF)
return result

end #def response

end #class ResponseParser

end #class IMAP

end #module Net

I've tested this 'monkeypatched' IMAP class long enough to say that fortunately this is all we need to make our client work... does it means that Microsoft is really shifting towards standards-based technology? Bah!

Real citizens of a virtual world or virtual citizens of a real world?

I like GPG.
When I started wandering around the Web with my modem about fifteen years ago, the impression was that of being a ghost. There were channels and newsgroups, but behind the words you wrote you could be anyone.
Internet was anonymous.
Time has passed, at the beginning we worried about not being anonymous anymore, then we started not wanting to be that anymore. The Internet has become a virtual extension of our social space.
With the first social networks we thought we could finally have the nationality of this virtual world, but once again we were wrong. We have become virtual citizens of a real world.
What we do on social networks has real world consequences because, in fact, the Internet IS the real world.
Every company has its own site, unique and recognizable. It is difficult for a phishing or cybersquatting not being soon discovered.
But, on the Internet we are less real than the Internet itself. Our virtual identities are ephemeral and too easy to counterfeit and violate. Anyone can pretend to be us, by registering on our behalf, robbing a password, or self attributing pictures, videos, comments or even entire blogs.
And increasingly those who do not know us personally take an idea of us with an online search.
Yet there is a standard protocol (RFC 4880 ), a standard as the email and the Internet itself, which guarantees to each of us a Pretty Good Privacy (PGP).

Each sysadmin knows and uses SSH. And being lazy as all the sysadmins has learned that he can store his public key on remote servers for not even having to type a password to connect.
Indeed, this mechanism should provide better security than passwords, but it's not true because nobody cares about the keys and keeps them safe. On the contrary, during hardware or software changes SSH keys are easily regenerated.

A sysadmin has learned that this message:

means to delete the corresponding line from ~ / .ssh / known_hosts
Almost all sysadmins that I know, at the sight of the message, not even go look for the line and delete the entire portfolio of keys.
And even the very rare cases of people who care about preserving and checking the keys, completely ignore their AUTHENTICITY.

GIT is a distributed code versioning system, and for programmers is a revolution. But lacking the central server repository as a guarantee of the revisions, disappears even the last glimmer of authenticity of the code.
So we put it all on GitHub ... but as long as we rely on the SSH only we are just at the same point.

GPG is not more complex than GIT. Those who keep care of their GPG keys why don't use this portfolio of keys for SSH?
GPG is powerful. Allows you to generate subkeys of the primary key (which should be kept on a disconnected storage and used only when necessary), to choose an expire date, and even to revoke them.
And unlike certificates is as reliable, is free and requires no bureaucratic times.
I have read of the possibility to export a GPG subkey and use it as public key for SSH, but publications on the correct procedure are scarce. Since when it comes to safety it's better not to improvise, I decided to ask for help from someone who was an expert rather than do it by myself. On StackOverflow the question was even banned from a security specialist arguing that it "solicited opinions, debates, discussions, surveys, or flaming".
Maybe no one really cares, because we all want to remain virtual citizens of a virtual world.



Read this post in italian