Power to the terminal command line: download any video by yourself

Those that, like me, don't have a flat rate for the Internet connection, know how careful you have to be watching to an interesting online video, like a video tutorial or a webinar, when the site doesn't allow you to review it offline, because if you have to watch it again later, your time/volume price limit approaches as quickly as larger is the video size.

However, the fact that a website does not allow the download of an online video is not quite true: your browser has to download it indeed, and it even temporarily stores it for the time that it needs it.
If this does surprise you, you might find useful to take a closer look at what does your browser when you click to watch a video online.

The video, like any other resource on the web page, has to be linked in the page your browser is showing: this means that parsing the code of the page it has to find its URL.
But if you look at the page source sometimes you are not able to find it, what you see instead is something like:

<script src="http://cssjs.chesscomfiles.com/js/videoplayer.js">
</script>
<script>
BuildSWFObject(592,'8917218|86291812c0025268a81210a7c89cfbac','modern-benoni-1-the-amanovs-thrilling-battle','videocontainer');
</script>

or

<embed allowscriptaccess="always" flashvars="file=http://www.chess.com/show/ 8917218&amp;autostart=false&amp;config=http://www.chess.com/config.xml" height="180" menu="false" pluginspage="http://www.macromedia.com/go/getflashplayer" quality="high" src="http://www.chess.com/player.swf" type="application/x-shockwave-flash" width="320"></embed>

This means the video location will be revealed dynamically when the video player's script is executed.

Using some tools to debug the browser behaviour, like Safari's Web Inspector, we can easily find the exact GET statement used to retrieve the video because most of the time it is the largest object in the page, so sorting the request by objects size or by loading time it will be the first result:

http://www.chess.com/video/serve-video?file=modern-benoni-1-the-amanovs-thrilling-battle&streamer=lighttpd&token=8917218|86291812c0025268a81210a7c89cfbac&crap=mp4?start=0&id=playerId&client=FLASH%20MAC%2011,1,102,55&version=4.3.132&width=710

Taking a closer look at this GET, we will see that a query is performed using the parameters given in the above script:

file: modern-benoni-1-the-amanovs-thrilling-battle
token: 8917218|86291812c0025268a81210a7c89cfbac

Besides, every GET declares a "User-Agent" and a "Referer Page", used by the server to know the kind of browser and where this request comes from.
If the site needs to know something more about who is making the request, it will expects a cookie, a small piece of text stored by the browser to uniquely identify the current user or browsing session. You can find out this cookie in the Web Inspector by matching the web site hostname:

Now you have all the information needed to ask by yourself the video to the web server, you just need some tool to manually send out this GET: wget or cURL, to name only the best known.

With curl these are the options to set the User-Agent, the Referer and the cookie:

-A/--user-agent
-b/--cookie (The data should be in the format "NAME1=VALUE1; NAME2=VALUE2")
-e/--referer
-o/--output Write output to instead of stdout.

By putting all together we can test if it works by downloading manually the file:

$ curl -e 'http://www.chess.com/video/player/modern-benoni-1-the-amanovs-thrilling-battle' -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.52.7 (KHTML, like Gecko) Version/5.1.2 Safari/534.52.7' -o test_GET.mp4 -b'__utmz=1.7324923106.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmc=1; __utmb=1.3.99.2139100016; __utma=1.721394078.1214124012.1214859116.8217491996.3; meebo-cim-session=3142ff122e11209bb12c; mcim=7281478shfjf8g1ASASGKJgakgaskg88&AgANGJJGAKKhagsjgasgkgkasgkkgahsAFSkgjj73263853930139501915hgn3031nc90000128tnguj%2hjfd9hgh32h88813jkgkgkkgk138; _chartbeat2=98124091hhhfaksn.1849812495199; ki_t=1324118915713%3B1324118915713%3B1324118915713%3B1%3B1; ki_u=ababa666-12de-5121-0241-fef88ccac10a; view_counter=hsahgk3r7881r7v318h13921r8; __qca=P0-841249119-8817214891285; cal=hdasjsfhjhjAJSFJHFAJ28389391239r9vkdjskfjkjfakj34h1jhjGKGKjkj3k12j3rkvjk13jrkvjk1j2kjgkjk21kjg8214781249fa99214899982191248SHHJEJBedbeuEBbeF; tf_login_id=xhejak; PHPSESSID=sakjgjk2383gakj3t98gj1jj11; __utmz=21314121.7381273182.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmc=56795640; __utmb=21314121.1.10.2138921496; __utma=12144141.521442111.1214148871.8912481811.1142819189.1; __utmxx=72121411.; __utmx=21314121.' 'http://www.chess.com/video/serve-video?file=modern-benoni-1-the-amanovs-thrilling-battle&streamer=lighttpd&token=8917218|86291812c0025268a81210a7c89cfbac&crap=mp4?start=0&id=playerId&client=FLASH%20MAC%2011,1,102,55&version=4.3.132&width=710'

% Total % Received % Xferd  Average Speed   Time    Time     Time  Current
                            Dload  Upload   Total   Spent    Left  Speed
5 22.8M 5 1182k    0     0  35135      0  0:11:20  0:00:34  0:10:46 12231^C

Keep in mind that the cookies and the token used might be useful only within the current browser session, so you can make the previous test only during the current session of the browser with the web site.

A good example of password security fail

Se la sicurezza si vede dal mattino, alla Regione Lombardia sono messi male.

Non sei ancora registrato che già ti mettono in guardia sulle loro competenze:

Buon cielo! Perché mai la mia password non dovrebbe avere due caratteri consecutivi uguali!?
Non staranno davvero dicendo che una password con due caratteri ripetuti è meno sicura delle altre...

Si potrebbe anche lasciare al buon senso stabilire se

"HHZ915ji"

è più indovinabile di

"PAPERINO"

ma almeno rendiamoci conto di quanto sia più facile per un password cracker listare le password valide o forzare la password di un account se queste non possono avere caratteri ripetuti!

Se n sono i caratteri ammissibili nella password, ci sono n^k possibili password di lunghezza k; ma ci sono solo n(n-1)^(k-1) possibili password senza caratteri consecutivi ripetuti, perché scelto arbitrariamente il primo carattere, devo scegliere il successivo tra gli N-1 caratteri diversi dall'ultimo digitato.

Questo, tradotto in numeri, significa che se le password mediamente sono di 12 caratteri su 37 (lettere, numeri e spazi), sapendo che non ci sono caratteri ripetuti il nostro cracker ha già scartato il 26% delle possibilità! Ha un quarto di possibilità in più di indovinare.
Anche per la minoranza di utenti che usano password con maiuscole e minuscole contemporaneamente le scelte sono comunque ridotte del 16%, oltre un sesto.

In generale su un alfabeto di 63 caratteri (lettere maiuscole e minuscole, numeri e spazi) la percentuale di password senza caratteri ripetuti diminuisce quasi linermente all'aumentare della lunghezza della password secondo questo grafico

Da cui vediamo che
Le password di 7 caratteri non ripetuti sono circa il 90% del totale
Le password di 15 caratteri non ripetuti sono circa l'80% del totale
Le password di 23 caratteri non ripetuti sono circa il 70% del totale
Le password di 32 caratteri non ripetuti sono circa il 60% del totale

Cioè se è vero che all'aumentare della lunghezza della password il cracker impiegherà più tempo, alla Regione Lombardia gli fanno uno sconto sempre più conveniente.

Tempi di saldi natalizi cari cracker! :)

Ruby how to get my private and public IP address

Ok,
you've just deployed your newest ruby app on a bunch of servers, and you need that this app knows the IP address of the server where it's running.

I've read some bizarre ways ("Get your local IP address" or "Get your local IP address") to do this, such as opening an UDP socket and inferring it from the interface used to route the packet.
With the Socket class you may do this more easily and also get the benefit of having useful Addrinfo objects and you are able to distinguish easily between public and private interfaces.

First of all:

Socket::ip_address_list

returns the Array of Addrinfo objects with all your interfaces (it deals with both IPv4 and IPv6).

You can then filter them using the standard Enumerable methods select() and detect() along with these Addrinfo methods:

Addrinfo#ipv4?
Addrinfo#ipv6?
Addrinfo#ipv4_private?
Addrinfo#ipv4_loopback?
Addrinfo#ipv4_multicast?

and convert them to string in dotted notation with

Addrinfo#ip_address

as in:

def all_my_ipv4_interfaces

socket.ip_address_list.select{|intf| intf.ipv4?}

end

def my_loopback_ipv4

socket.ip_address_list.detect{|intf| intf.ipv4_loopback?}

end

def my_first_private_ipv4

socket.ip_address_list.detect{|intf| intf.ipv4_private?}

end

def my_first_public_ipv4

socket.ip_address_list.detect{|intf| intf.ipv4? and !intf.ipv4_loopback? and !intf.ipv4_multicast? and !intf.ipv4_private?}

end

all_my_ipv4_interfaces

=> [#<Addrinfo: 127.0.0.1>, #<Addrinfo: 10.212.142.73>, #<Addrinfo: 62.127.4.80>]

my_first_private_ipv4

=> #<Addrinfo: 10.212.142.73>

my_first_public_ipv4.ip_address

=> "62.127.4.80"

Real citizens of a virtual world or virtual citizens of a real world?

I like GPG.
When I started wandering around the Web with my modem about fifteen years ago, the impression was that of being a ghost. There were channels and newsgroups, but behind the words you wrote you could be anyone.
Internet was anonymous.
Time has passed, at the beginning we worried about not being anonymous anymore, then we started not wanting to be that anymore. The Internet has become a virtual extension of our social space.
With the first social networks we thought we could finally have the nationality of this virtual world, but once again we were wrong. We have become virtual citizens of a real world.
What we do on social networks has real world consequences because, in fact, the Internet IS the real world.
Every company has its own site, unique and recognizable. It is difficult for a phishing or cybersquatting not being soon discovered.
But, on the Internet we are less real than the Internet itself. Our virtual identities are ephemeral and too easy to counterfeit and violate. Anyone can pretend to be us, by registering on our behalf, robbing a password, or self attributing pictures, videos, comments or even entire blogs.
And increasingly those who do not know us personally take an idea of us with an online search.
Yet there is a standard protocol (RFC 4880 ), a standard as the email and the Internet itself, which guarantees to each of us a Pretty Good Privacy (PGP).

Each sysadmin knows and uses SSH. And being lazy as all the sysadmins has learned that he can store his public key on remote servers for not even having to type a password to connect.
Indeed, this mechanism should provide better security than passwords, but it's not true because nobody cares about the keys and keeps them safe. On the contrary, during hardware or software changes SSH keys are easily regenerated.

A sysadmin has learned that this message:

means to delete the corresponding line from ~ / .ssh / known_hosts
Almost all sysadmins that I know, at the sight of the message, not even go look for the line and delete the entire portfolio of keys.
And even the very rare cases of people who care about preserving and checking the keys, completely ignore their AUTHENTICITY.

GIT is a distributed code versioning system, and for programmers is a revolution. But lacking the central server repository as a guarantee of the revisions, disappears even the last glimmer of authenticity of the code.
So we put it all on GitHub ... but as long as we rely on the SSH only we are just at the same point.

GPG is not more complex than GIT. Those who keep care of their GPG keys why don't use this portfolio of keys for SSH?
GPG is powerful. Allows you to generate subkeys of the primary key (which should be kept on a disconnected storage and used only when necessary), to choose an expire date, and even to revoke them.
And unlike certificates is as reliable, is free and requires no bureaucratic times.
I have read of the possibility to export a GPG subkey and use it as public key for SSH, but publications on the correct procedure are scarce. Since when it comes to safety it's better not to improvise, I decided to ask for help from someone who was an expert rather than do it by myself. On StackOverflow the question was even banned from a security specialist arguing that it "solicited opinions, debates, discussions, surveys, or flaming".
Maybe no one really cares, because we all want to remain virtual citizens of a virtual world.



Read this post in italian

Add a shortcut to TextMate to lookup a class or method definition in a tooltip

UPDATED Mar 11 Ott 2011 12:52:36 CEST

I'm Vim user. I love the command mode, and I had fun to dance with those keystroke commands.
However changing OS means viewing things in another perspective, and switching from Linux to Mac OS for me meant choosing an editor that has its roots in the operating system where it runs.
In the case of Mac OS it not only means it should have a suitable GUI, but it should fire up instantly and above all it should behave like a Mac App.
TextMate was the obvious choice. Because behaving like a Mac App means that if you pick a folder in the Finder and drop it in the editor icon, it should straight open it like a Project. And it means that under the guise of a bare and simple application hides a world of possibilities and customization. That world in TextMate is called Bundles.

Let's talk about Bundles with a real case. (TextMate|I) was missing a shortcut to quickly lookup the definition of a class or method name, so I decided to write a TextMate command that shows me the definition in a tooltip for the current word under the caret.
The difficulties I had were merely related to the complexity of this shell scripting. Since I was trying to keep it as simple and efficient I could, I chose to use the standard unix tools: grep find and sed (it ended up that maybe a ruby script would have done it easier :))

All you have to do to use it in your TextMate is to open the Bundle editor (⌃ ⌥ ⌘ B), make a new command under "Ruby", assign a "Key Equivalent" like ⌃ ] and paste the following script in the "Command(s):" windows

#!/bin/bash
SEARCHPATH=$(test -x "$TM_PROJECT_DIRECTORY" && echo "$TM_PROJECT_DIRECTORY" || echo $(dirname "$TM_FILEPATH") )
STARTPATTERN="^([[:space:]]|;)*(def|class)([[:space:]]|;)+$TM_CURRENT_WORD([^A-Za-z0-9_].*)*$"
ENDPATTERN="^([[:space:]]|;)*end([[:space:]]|;)*"
FILESMATCH=$( (egrep -ns "$STARTPATTERN" "$TM_FILEPATH" || egrep -nsR "$STARTPATTERN" --exclude="*.svn*" "$SEARCHPATH") |cut -d: -f-2 )
echo $FILESMATCH # Show on tooltip
echo $FILESMATCH |cut -d: -f1 |xargs -0 basename |pbcopy # Copy on clipboard for later use with cmd+T
find "$TM_FILEPATH" "$SEARCHPATH" -type f -name '*.rb' -exec sed -En "/$STARTPATTERN/,/$ENDPATTERN/p" {} \; |sed -E "/$ENDPATTERN/q"

CODE UPDATED Mar 11 Ott 2011 12:52:36 CEST
Now it also copies the first matching filename to the clipboard, so you can open this file by pressing cmd+T (and then go to the exact line with cmd+L)

Make sure the output is "Show as Tool Tip" like this:

If you've done it correctly, placing the caret on or after any method or class name and pressing ⌃ ] (or whatever key assignment you've made) will search in the current file ($TM_FILEPATH) and then recursively in the project directory ($TM_PROJECT_DIRECTORY) or in the file directory (if file is not in a project), and show a tooltip with:

1. the name of the file (if different from the one you are editing) and the line where the first definition of the method or class is found
2. the portion of definition from the declaration to the first "end" found

Although not perfect, it will do its best to avoid commented lines and false matches.
Even if this script was done for ruby, you can use it as a base for other programming languages, just change the -name '*.rb' part of the find command in the extension of your sources, and the keywords def - class - end that matches your method, function or class definitions.

In the very unsual case you are running TextMate not on a Mac OS, or if you want to adapt the script to run in a linux shell, just skip the line to "Copy on clipboard for later use with cmd+T" and remember that to use the Extended Regular Expression in SED you have to use the -r switch instead of -E.

For any explanation, contribution or discussion on the script please leave a comment below, and if you find it useful please leave me a positive feedback! :)

My CSS stylesheet for Ruby on Rails syntax highligthing

As my very first post, I would like to share with you the CSS stylesheet I use to format Ruby snippets.
If you need a CSS stylesheet ready to be applied to your html code block, or you simply like my ruby formatting color schema, you can take it!

div.rb-code {
 padding:1em 1em 1em 1em;
 border:0.2em groove #000000;
 font-family:Andale Mono,monospace;
 font-size:1.2em;
 color:#CC9900;
 background-color:#3A3A3A;
 overflow:auto;
 z-index:1;
}
div.rb-block-def {}
div.rb-block-exe {}
div.rb-code div.tab { padding-left:1.4em; }
div.rb-code .attribute { color:#FFFF66; }
div.rb-code .argument { color:#FFFF00; font-style:italic; }
div.rb-code .class-parent { color:#00FF00; }
div.rb-code .class { color:#FF00FF; font-weight:bold; }
div.rb-code .comment { color:#FFFFFF; }
div.rb-code .constant { color:#FF9966; }
div.rb-code .keyword { color:#FF8000; text-decoration:underline; }
div.rb-block-def .method { color:#FF00FF; }
div.rb-code .module { color:#A020F0; font-weight:bold; }
div.rb-code .number { color:#00FFFF; font-weight:bold; }
div.rb-code .regexp { color:#FF0000; }
div.rb-code .string { color:#99CCFF; }
div.rb-code .symbol { color:#CCFF66; }
div.rb-code .var { color:#FFFF66; font-weight:bold; }
div.rb-block-exe .command { color:#99CCFF; font-weight:bold; }
div.rb-block-exe .evaluation { color:#F0F0F0; font-size:0.8em; }
div.rb-block-exe .prompt { color:#CC9900; }

To use it, simply place all your code snippet between <DIV class='rb-code'>...</DIV>, and then wrap inside the appropriate <SPAN>...</SPAN> block the portions you need to highligth.

You can distinguish def blocks and executable blocks placing them inside a <DIV class='rb-block-def'> or a <DIV class='rb-block-exe'>

To ident a block, put it inside another <DIV class='tab'>

Look at the following example:

<DIV class='rb-code'>  
<DIV class='rb-block-def'>
<DIV><SPAN class='keyword'>class</SPAN> <SPAN class='class'>Person</SPAN></DIV>
<DIV class='tab'>
<DIV><SPAN class='keyword'>def</SPAN> <SPAN class='method'>speak_now</SPAN>(<SPAN class='argument'>thought</SPAN>)</DIV>
<DIV class='tab'>
<DIV>puts <SPAN class='var'>thought</SPAN></DIV>
</DIV>
<DIV><SPAN class='keyword'>end</SPAN></DIV>
</DIV>
<DIV><SPAN class='keyword'>end</SPAN></DIV>
</DIV>
</DIV>

it will be shown as:

class Person

def speak_now(thought)

puts thought

end

end