Power to the terminal command line: download any video by yourself

Those that, like me, don't have a flat rate for the Internet connection, know how careful you have to be watching to an interesting online video, like a video tutorial or a webinar, when the site doesn't allow you to review it offline, because if you have to watch it again later, your time/volume price limit approaches as quickly as larger is the video size.

However, the fact that a website does not allow the download of an online video is not quite true: your browser has to download it indeed, and it even temporarily stores it for the time that it needs it.
If this does surprise you, you might find useful to take a closer look at what does your browser when you click to watch a video online.

The video, like any other resource on the web page, has to be linked in the page your browser is showing: this means that parsing the code of the page it has to find its URL.
But if you look at the page source sometimes you are not able to find it, what you see instead is something like:

<script src="http://cssjs.chesscomfiles.com/js/videoplayer.js">
</script>
<script>
BuildSWFObject(592,'8917218|86291812c0025268a81210a7c89cfbac','modern-benoni-1-the-amanovs-thrilling-battle','videocontainer');
</script>

or

<embed allowscriptaccess="always" flashvars="file=http://www.chess.com/show/ 8917218&amp;autostart=false&amp;config=http://www.chess.com/config.xml" height="180" menu="false" pluginspage="http://www.macromedia.com/go/getflashplayer" quality="high" src="http://www.chess.com/player.swf" type="application/x-shockwave-flash" width="320"></embed>

This means the video location will be revealed dynamically when the video player's script is executed.

Using some tools to debug the browser behaviour, like Safari's Web Inspector, we can easily find the exact GET statement used to retrieve the video because most of the time it is the largest object in the page, so sorting the request by objects size or by loading time it will be the first result:

http://www.chess.com/video/serve-video?file=modern-benoni-1-the-amanovs-thrilling-battle&streamer=lighttpd&token=8917218|86291812c0025268a81210a7c89cfbac&crap=mp4?start=0&id=playerId&client=FLASH%20MAC%2011,1,102,55&version=4.3.132&width=710

Taking a closer look at this GET, we will see that a query is performed using the parameters given in the above script:

file: modern-benoni-1-the-amanovs-thrilling-battle
token: 8917218|86291812c0025268a81210a7c89cfbac

Besides, every GET declares a "User-Agent" and a "Referer Page", used by the server to know the kind of browser and where this request comes from.
If the site needs to know something more about who is making the request, it will expects a cookie, a small piece of text stored by the browser to uniquely identify the current user or browsing session. You can find out this cookie in the Web Inspector by matching the web site hostname:

Now you have all the information needed to ask by yourself the video to the web server, you just need some tool to manually send out this GET: wget or cURL, to name only the best known.

With curl these are the options to set the User-Agent, the Referer and the cookie:

-A/--user-agent
-b/--cookie (The data should be in the format "NAME1=VALUE1; NAME2=VALUE2")
-e/--referer
-o/--output Write output to instead of stdout.

By putting all together we can test if it works by downloading manually the file:

$ curl -e 'http://www.chess.com/video/player/modern-benoni-1-the-amanovs-thrilling-battle' -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.52.7 (KHTML, like Gecko) Version/5.1.2 Safari/534.52.7' -o test_GET.mp4 -b'__utmz=1.7324923106.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmc=1; __utmb=1.3.99.2139100016; __utma=1.721394078.1214124012.1214859116.8217491996.3; meebo-cim-session=3142ff122e11209bb12c; mcim=7281478shfjf8g1ASASGKJgakgaskg88&AgANGJJGAKKhagsjgasgkgkasgkkgahsAFSkgjj73263853930139501915hgn3031nc90000128tnguj%2hjfd9hgh32h88813jkgkgkkgk138; _chartbeat2=98124091hhhfaksn.1849812495199; ki_t=1324118915713%3B1324118915713%3B1324118915713%3B1%3B1; ki_u=ababa666-12de-5121-0241-fef88ccac10a; view_counter=hsahgk3r7881r7v318h13921r8; __qca=P0-841249119-8817214891285; cal=hdasjsfhjhjAJSFJHFAJ28389391239r9vkdjskfjkjfakj34h1jhjGKGKjkj3k12j3rkvjk13jrkvjk1j2kjgkjk21kjg8214781249fa99214899982191248SHHJEJBedbeuEBbeF; tf_login_id=xhejak; PHPSESSID=sakjgjk2383gakj3t98gj1jj11; __utmz=21314121.7381273182.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmc=56795640; __utmb=21314121.1.10.2138921496; __utma=12144141.521442111.1214148871.8912481811.1142819189.1; __utmxx=72121411.; __utmx=21314121.' 'http://www.chess.com/video/serve-video?file=modern-benoni-1-the-amanovs-thrilling-battle&streamer=lighttpd&token=8917218|86291812c0025268a81210a7c89cfbac&crap=mp4?start=0&id=playerId&client=FLASH%20MAC%2011,1,102,55&version=4.3.132&width=710'

% Total % Received % Xferd  Average Speed   Time    Time     Time  Current
                            Dload  Upload   Total   Spent    Left  Speed
5 22.8M 5 1182k    0     0  35135      0  0:11:20  0:00:34  0:10:46 12231^C

Keep in mind that the cookies and the token used might be useful only within the current browser session, so you can make the previous test only during the current session of the browser with the web site.

Add a shortcut to TextMate to lookup a class or method definition in a tooltip

UPDATED Mar 11 Ott 2011 12:52:36 CEST

I'm Vim user. I love the command mode, and I had fun to dance with those keystroke commands.
However changing OS means viewing things in another perspective, and switching from Linux to Mac OS for me meant choosing an editor that has its roots in the operating system where it runs.
In the case of Mac OS it not only means it should have a suitable GUI, but it should fire up instantly and above all it should behave like a Mac App.
TextMate was the obvious choice. Because behaving like a Mac App means that if you pick a folder in the Finder and drop it in the editor icon, it should straight open it like a Project. And it means that under the guise of a bare and simple application hides a world of possibilities and customization. That world in TextMate is called Bundles.

Let's talk about Bundles with a real case. (TextMate|I) was missing a shortcut to quickly lookup the definition of a class or method name, so I decided to write a TextMate command that shows me the definition in a tooltip for the current word under the caret.
The difficulties I had were merely related to the complexity of this shell scripting. Since I was trying to keep it as simple and efficient I could, I chose to use the standard unix tools: grep find and sed (it ended up that maybe a ruby script would have done it easier :))

All you have to do to use it in your TextMate is to open the Bundle editor (⌃ ⌥ ⌘ B), make a new command under "Ruby", assign a "Key Equivalent" like ⌃ ] and paste the following script in the "Command(s):" windows

#!/bin/bash
SEARCHPATH=$(test -x "$TM_PROJECT_DIRECTORY" && echo "$TM_PROJECT_DIRECTORY" || echo $(dirname "$TM_FILEPATH") )
STARTPATTERN="^([[:space:]]|;)*(def|class)([[:space:]]|;)+$TM_CURRENT_WORD([^A-Za-z0-9_].*)*$"
ENDPATTERN="^([[:space:]]|;)*end([[:space:]]|;)*"
FILESMATCH=$( (egrep -ns "$STARTPATTERN" "$TM_FILEPATH" || egrep -nsR "$STARTPATTERN" --exclude="*.svn*" "$SEARCHPATH") |cut -d: -f-2 )
echo $FILESMATCH # Show on tooltip
echo $FILESMATCH |cut -d: -f1 |xargs -0 basename |pbcopy # Copy on clipboard for later use with cmd+T
find "$TM_FILEPATH" "$SEARCHPATH" -type f -name '*.rb' -exec sed -En "/$STARTPATTERN/,/$ENDPATTERN/p" {} \; |sed -E "/$ENDPATTERN/q"

CODE UPDATED Mar 11 Ott 2011 12:52:36 CEST
Now it also copies the first matching filename to the clipboard, so you can open this file by pressing cmd+T (and then go to the exact line with cmd+L)

Make sure the output is "Show as Tool Tip" like this:

If you've done it correctly, placing the caret on or after any method or class name and pressing ⌃ ] (or whatever key assignment you've made) will search in the current file ($TM_FILEPATH) and then recursively in the project directory ($TM_PROJECT_DIRECTORY) or in the file directory (if file is not in a project), and show a tooltip with:

1. the name of the file (if different from the one you are editing) and the line where the first definition of the method or class is found
2. the portion of definition from the declaration to the first "end" found

Although not perfect, it will do its best to avoid commented lines and false matches.
Even if this script was done for ruby, you can use it as a base for other programming languages, just change the -name '*.rb' part of the find command in the extension of your sources, and the keywords def - class - end that matches your method, function or class definitions.

In the very unsual case you are running TextMate not on a Mac OS, or if you want to adapt the script to run in a linux shell, just skip the line to "Copy on clipboard for later use with cmd+T" and remember that to use the Extended Regular Expression in SED you have to use the -r switch instead of -E.

For any explanation, contribution or discussion on the script please leave a comment below, and if you find it useful please leave me a positive feedback! :)