4chan download script

Sunday, 31. August 2008 19:44 - daniel - Other - 20 Comments


Downloads all the images from a 4chan image thread. I will probably regret downloading anything from 4chan, but that's not my problem.

Usage: 4chandl <4chan thread url>

Download the script

#!/bin/sh

if [ "$1" = "" ]; then
	echo "Usage: `basename $0` <4chan thread url>"
	exit 1
fi

echo "4chan downloader"
echo "Downloading untill canceled or 404'd"
LOC=$(echo "$1" | egrep -o '([0-9]*).html' | sed 's/\.html//g' )
echo "Downloading to $LOC"

if [ ! -d $LOC ]; then
	mkdir $LOC
fi

cd $LOC

while [ "1" = "1" ]; do
	TMP=`mktemp`
	TMP2=`mktemp`

	wget -O "$TMP" "$1"
	if [ "$?" != "0" ]; then
		rm $TMP $TMP2
		exit 1
	fi

	egrep 'http://(img|cgi).4chan.org/[a-z0-9]+/src/(cb-nws/)?([0-9]*).(jpg|png|gif)' "$TMP" -o > "$TMP2"
	cat "$TMP2" | sed 's!/cb-nws!!g' > "$TMP"

	wget -nc -i $TMP
 
	rm $TMP $TMP2
 
	echo "Waiting 30 seconds befor next run"
	sleep 30
done;



Frea, Champion of the Naaru

Saturday, 30. August 2008 16:34 - daniel - World of Warcraft - 0 Comments




Yesterday I got the title Champion of the Naaru for my little druid.



Getting the title is pretty hard, considering I had to tank all the heroic instances with only a few epics and not that much tanking experience. Especially the Shattered Halls and Shadow Labyrinth runs took us a long time.

Still, I think the title is worth the effort.



New WotLK Build (Update)

Saturday, 30. August 2008 1:41 - daniel - World of Warcraft - 0 Comments




Blizzard updated WotLK to 3.0.2.8885 today. This was the first test under real conditions for my automatic update system for my Wrath of the Lick King beta patch mirror.

What should I say? It worked nearly flawless. The next update can come even without me :D.

Also, I expect traffic for this month to be around 700 to 800 GB. I hope that it will go down a bit in September, or I have to take the mirror offline after I hit the traffic cap.

Update
The server apparently crashed at 11:20 today. I guess the connections from CoBlitzProxy are too much for Apache to handle. I mean, we are talking about 30 or more connections per second at peak times. The major cause was probably too many apache processes running, taking up all the memory.

Anyway, I tuned Apache a bit to use less memory, but that only helped a bit. The load was still very high. I was probably only a matter of time until the server crashes again.

Because of this I installed lighty. I hope lighty will do better than apache and handles those connections without using too much CPU power or memory.



WotLK mirror

Monday, 25. August 2008 11:33 - daniel - Other - 0 Comments




I'm going to be away for the next 6 months, leaving me without internet access during the weekdays and sometimes on the weekends too. To keep my WotLK mirror up to date during this time, I created a collections of scripts to automatically download the patches and update the mirror site.

In order to do this, you have to know how Blizzard updates World of Warcraft. The analysis on Network Uptime helped me quite a bit here. Basically, the WoW Launcer checks a .txt file on the Blizzard server if there is a update available for the current version. If that's the case, it downloads a compressed archive containing the update downloader, a file with instructions and 2 more files.

The file is compressed with Blizzards mpq format and that's where my problems started. I tested the mpq-tools from babelize.org on my local box and everything was fine. But on the server (running Debian) I wasn't able to compile them. Using the latest revision from svn worked to compile the libmpq. But the mpq-tools failed while creating the configure script because there was a package missing. Anyway, thanks to the guys on #mpq-tools @ irc.freenode.net.

Well, after the downloader was extracted from the file it is pretty easy to get the .torrent file from it. I use the perl script from  Matt Francis for this.

Downloading the torrents is a bit more difficult. The rtorrent version in debian stable can watch a directory for new torrent files and start the automatically. But it can't move the files to a different location afterwards, nor show the download status. Downloading to the patch mirrors directory and making incomplete files available didn't sound like a good idea. Because of this I had to compile the latest version of rtorrent and libtorrent.

Well, all of this took quite some time, but now the mirror should stay up to date even without me :D.



WoW: Wrath of the Lich King

Monday, 11. August 2008 23:56 - daniel - World of Warcraft - 0 Comments




I got a beta key a few days ago. I haven't really been able to play very much yet because  the server is unstable and buggy. Crashes occur quite often, and there was an issue with loot laging for minutes every time you got a new item. Anyway, I hope I'm able to play the new content (which is looking great thereby) when the server runs more smoothly again.

In the mean time, I created a WotLK beta patch mirror. When I downloaded and installed the beta client and all the updates, I wasn't able to find a patch mirror and the blizzard downloader is not the fastest to say the least. Therefore I created this site to help people get the new patches.



Newer
Older