Filed under: Linux | bbot's blog

copper II: apache virtual host configuration

bbot — 2009-07-25T23:30:16-05:00

Blogathon 2009 wordcount: 907

So I purchased a dedicated server from serverpronto (the starter+) on advice of Steve, because I was tired of using NFSnet's e-mail forwarding to a gmail account, since that toolchain relies on several people to do their job perfectly, one of which (Google) who doesn't even get paid for privilege. I'd much rather have the whole mess in my hands, so I can get mad at myself when it doesn't work.

But configuring postfix is hard, and apache is easy, so I decided to get apache working first.

The server itself I named copperII, in memory of the late, great, copper.

I have a total of five domains, bbot.org, 7chon.org, unitinu.net, kyonko.org and bierwagen.us. The last four don't get any traffic that matters, so I moved them over first, saving bbot.org for last.

Moving was a simple matter of four ftp operations, and one scp. The Pointless Four consist of either a single index.html file each, (7chon, bierwagen) or an index and an image. (kyonko, unitinu)

bbot.org, of course, has quite a lot more stuff, and thus required a recursive scp. Amusingly, this had to be done from copperII, and not from my laptop, since a architectural bug requires a fairly insecure workaround to be able to ssh out from ssh.phx.nearlyfreespeech.net. Bizarrely, this presents with the error message "Host key verification failed", when the problem is that it can't open /dev/tty.

I copied each site to its own directory under /srv/www/vhosts/, then proceeded to make a virtual host configuration file for each site in /etc/apache2/vhosts.d/ by editing the provided template file with the information for 7chon.org, then using a simple sed pipe operation of the form "s/target/replacement/g". S means substitute, and the final flag means "globally", or else sed will just replace the first instance of target with replacement, which is hilariously confusing the first time you forget the final g.

copperII:/etc/apache2/vhosts.d # sed s/7chon/unitinu/g 7chon.conf > unitinu.conf copperII:/etc/apache2/vhosts.d # sed s/org/us/g unitinu.conf

The format of the virtual host config file is fairly straightforward. For minimum functionality, all you have to do it set ServerName to the domain name, then DocumentRoot and the final Directory tag to the full directory path. Once you do that, run httpd2 -t to check the configuration files (it should return "Syntax OK") and whatever distro-specific services manager operation to restart apache. In suse, it's /etc/rc.d/apache2 restart, though you should call it with status first to make sure it's running.

After that all you need to do is change the A records for each domain to point at the IP address of the server and it magically serves different content depending on what the client asks for! Magic!

On copper I set up logging so that bbot.org got its own log file, while the other domains logged to a common file. This is done by setting the CustomLog variable to /path/to/log and type of log, the type of log, if not one of the default log formats, being defined by LogFormat.

bbot.org was told to log to /var/log/apache2/bbot-access and bbot-error using the default combined log format, but the other sites had to use a custom log format that included the name of the site in the common virtual host log file, or else I would get confused, and cry. Fortunately for me, Apache provided an example of a common vhost log format in the documentation, saving me from having to exercise independent thought.

LogFormat "%v %l %u %t \"%r\" %>s %b" commonvhost ErrorLog /var/log/apache2/vhosts-error CustomLog /var/log/apache2/vhosts-access comonvhost

This started logging to the correct location, but all it actually logged was "comonvhost" over and over, indicating that it had correctly parse the configuration file, but had ignored the LogFormat setting, for some reason. You should be able to see why.

Correct! As stated in the Official Apache Foundation Documentation for HTTPd 2.0, there is a typo in the CustomLog setting, specifying the log format as "comonvhost" instead of "commonvhost". Apache, slavishly obeying what I told it to do instead of what I wanted it to do, proceeded to log the word "comonvhost" to the access log on every access. Thanks, Apache.

This can be thought of as an example of the age old SQL injection exploit, only implemented in brain matter, and thus impossible to patch. I implicitly trusted the documentation, and thus copy and pasted the code snippet into my configuration file without thinking about it. Thanks, brain.

Next: Awstats, Postfix, and the TF2 Dedicated Server!

fun with wget

bbot — 2009-06-19T18:11:39-05:00

So one Matt Blind, author of an excellent piece on bookstore customers also happens to produce bestseller lists for graphic novels. How? Well, in his own words:

The Core of the Charts is made up of data from three sites: Amazon, Barnes & Noble, and Borders.

Once a week, I visit each site to check their Graphic Novel categories, and I sort the search results by "bestselling". The links above will pull up exactly that.

I then click through, page after page, and type the titles into a spreadsheet in the order that they are ranked on the sales site. [this is the hard part]

And once I have a full list, I assign points to the books depending on how highly they rank. Add up the points each title earns (and add on similar data from a half-dozen second-tier sales sites) to get a composite score, and there's your ranking.

In concept, it's that simple.

In practice, because the sites themselves can update as often as once an hour, after I load up a website & sort the search results, I then click open each new page in a new tab until I have 20-100 tabs open, representing a snapshot of the full sales (top 900-1200 titles) of this particular sales site over a relatively short time-frame (10-15 minutes). And then I start the data entry.

Wow, that sucks. What did Raymond say about data entry?

Hackers (and creative people in general) should never be bored or have to drudge at stupid repetitive work, because when this happens it means they aren't doing what only they can do -- solve new problems. This wastefulness hurts everybody. Therefore boredom and drudgery are not just unpleasant but actually evil.

You're right, Raymond! This sorely needs some automation.

But complete automation would be Nontrivial, so lets take the merest of first steps. Opening hundreds of tabs sucks. How about just one tab?

First thing, I wrote a little bash script to generate the links list, since it would, of course, be 101 lines long, and I didn't feel like copy and pasting each one from firefox, or whatever.

First, let's look at the base url.

http://browse.barnesandnoble.com/browse/nav.asp?No=10&N=0+989443&Ne=989443&visgrp=fiction&act=BC_ANC

Clicking on "next" results in:

http://browse.barnesandnoble.com/browse/nav.asp?No=20&N=0+989443&Ne=989443&visgrp=fiction&act=BC_ANC

The string of interest is "No=20", which increments by ten every page. So the resulting script is:

for (( i=30; i <= 1000; i=i+10  ))

do

	echo "http://browse.barnesandnoble.com/browse/nav.asp?No=$i&N=0+989443&Ne=989443&visgrp=fiction&act=BC_ANC"

done

This script creates a variable (i), sets it to 30 (i=30), tells it to run the loop as long as i is less than or equal to 1000 (i <= 1000), then increments i by 10 each loop. (i=i+10). It then inserts the current value of i in the base url. It starts at 30 because I already had the first three pages of sales ranks.

Running this script and piping the output to "links.txt" results in a 101 line long text file, with a url on each line. We tell wget to use this file, to wait a random time between 0 and 2 seconds between hitting each link, and to use the user-agent "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618)", who happened to be the most recent visitor to bbot.org when I plundered the log for user agents. We also tell it to log to wget.log, and to be verbose, because --verbose is awesome.

wget --random-wait --user-agent="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618)" --input-file=links.txt --output-file=wget.log --verbose

Wget then hogs the terminal, but remains silent, because we told it to output to wget.log. We can watch along in another terminal by tail -f'ing wget.log, and occasionally ls'ing. It goes pretty fast, taking 20.52 seconds to download 7.4 mebibytes worth of html. After cat'ing everything you end up with a mighty browser-crushing 7.4 mebibyte html that makes Opera colossally unhappy, but which Firefox can handle fine-ish, and takes forever to upload, so we bzip it to a svelte 220 kibibytes.

http://bbot.org/sales.bz2

Standard large html file disclaimers apply. It's big big big, media rich, and outrageously malformed.

distro paranoia

bbot — 2009-04-28T16:08:24-05:00

So, it turns out that Paul Frields, project leader for Fedora Linux used to develop spyware for the FBI.

Well golly, I use Fedora.

Oh come on. The FBI doesn't care about linux users.

You're right, I'm sure the FBI has absolutely no interest in a group of technical experts whose work underlie critical national infrastructure and who refuse to use commerical software due to ideological reasons.

These people are beyond suspicion, while senators and charities are not.

Uh huh. Sure.

So I'm looking for a new distro. The limitations is that it has to be linux-based, and come with a package manager. What are my choices?

Ubuntu. PRO: Easily the most popular linux distro. CON: It's Ubuntu.
Debian. PRO: Ubuntu, but with less lameness. CON: Extremely ideological. Getting the NVidia drivers and mp3 support working will require effort. On the one hand, I had to do the same with Fedora. On the other, it's fucking 2009. I shouldn't have to do that anymore.
Gentoo. PRO: Nerdy. Compiling everything results in a faster, more optimized system, and also prevents attackers from shipping compromised binaries. CON: I'm not going to be reading the source code, either, so compiling everything isn't actually any more secure. Also, while the end result may be faster, I'll still be compiling everything on a 900mhz machine. Also also, it's gentoo, the distro of choice for people upgrading from ubuntu.
SuSE. PRO: I've actually used suse before, way back in middle school. Presumably the installer's better now. CON: Primarily a german-language distro. Documentation in english may be spotty.
Mandriva. PRO: It's a linux distro? CON: I've never used it, nor heard much of anything about it. Apparently it's popular enough to have a package manager.
Slackware. PRO: Pretty cool bros only. CON: I'm not all that cool of a bro. Plus, I like wussy window managers like GNOME and KDE, and the other slackware users might make fun of me for using them. Also, package selection may be weak or out of date, since the slackware maintainers are too busy telling kids to get off of their lawn to update their repositories.

Exciting!

uptime, how I shall miss you

bbot — 2008-04-17T17:22:01-05:00

I just restarted my computer for the first time in 149 days.

A moment of silence for my dear departed uptime.

from hell's heart I stab at thee, vi

bbot — 2008-04-13T14:40:32-05:00

So the previous blog entry looks pretty simple, right?

Wrong!

You see, I pasted the chat transcript directly into vi, which nanoblogger uses to edit blog entires. Nanoblogger uses direct html formatting, so the greater than/lesser than names were interpreted as tags, and thus didn't render. Lame!

HTML being a proper markup language, however, implements a kinda sorta escape code sequence, where you can use "<" and ">" (lesser-than and greater-than, get it?) to render < and > without your html interpreter (read: web browser) invisibly erasing them.

But the problem was, I had about twelve instances of < and > to replace. Text navigation in vi, while not impossibly difficult, is still awkward; and in any event, doing all that crap by hand was ideologically impure. Whatever was I to do?

Search and replace, of course!

Vi, being a Real Man's text editor, uses sed-style regex expressions for search and replace, and as I was about to discover, sed-style wildcards. The command string I first tried using was "%s//<bbot>/g" (%s/ to search every line, /g to replace every instance. The regex is of form /search string/thing to replace search string with/)

That command, of course, resulted in "lt;bbotgt;" instead of "", since motherfucking sed uses & in the replacement string to mean what the search string matched. This lets you do all sorts of enormously clever things, but is a giant pain in the fucking ass when you're messing with HTML enity codes.

Now, normally, you would use shell style quoting to tell sed to ignore &, but vi isn't the shell! It ignores quotes! I, of course, only figured this out after running through every permutation I could think of, before just using the \ escape character.

Builds character, right?

tor

bbot — 2008-04-07T12:40:51-05:00

So copper.unitinu.net is now running a Tor relay. Since there's nothing else running on that box, Tor is happily soaking up ten megabits a second. How did I do it, you ask?

Easy!

First I downloaded the tor and libevent sources.

[root@copper ~]#wget http://www.monkey.org/~provos/libevent-1.3e.tar.gz output removed [root@copper ~]#wget http://www.torproject.org/dist/tor-0.1.2.19.tar.gz output removed

Then I unpacked the archives, ran the makefiles, etc. But tor errored out during configuration!

checking whether we need extra options to link libevent... configure: error: Found linkable libevent in (system), but it doesn't run, even with -R. Maybe specify another using --with-libevent-dir? [root@copper tor-0.1.2.19]#

Since libevent installed correctly, I guessed that tor couldn't find the libraries. Googling the error message googlewhacked on another fellow with the same problem, which he solved by setting LD_LIBRARY_PATH to /usr/local/lib. So I, you know, did that.

[root@copper tor-0.1.2.19]# export LD_LIBRARY_PATH=/usr/local/lib output removed [root@copper tor-0.1.2.19]# ./configure && make output removed, but it worked! too bad make doesn't give a completion message.

That was easy! Kinda.

blackberry, why must you bedevil me so

bbot — 2008-03-11T08:36:05-05:00

So, my blackberry comes with a neat little utility called "memos", which, oddly enough, allows you to write memos. I've been using it to write bits and pieces of various things, and it finally came time that I had enough that I wanted to actually export them in a usable format. How is this done, you ask? Easy!

You back up the device using RIM's terrible backup utility. This takes an oddly long time, seeing as how it's only backing up text, and over a USB2 connection at that.
This produces an .ipd file, which is some wacky proprietary binary format, and cannot be read by any linux utility.
Find a program which claims to be able to read .ipd files; which is of course shareware, since it targets a tiny user base of people with too much money.
Download the free version, to discover that only lets you export ten memos at a time, because the cocksucker who wrote it wants twenty bucks.
Produce four .txt files of ten memos each, by hand, taking an inordinate amount of time.
Copy this over samba to the linux computer.
Use cat to mash the files together, then manually, again, paste each one into Tomboy

EASY

lol sl

bbot — 2006-10-15T22:59:24-05:00

2006-10-15T22:48:43Z WARNING: Alert: A new version of Second Life is available.
2006-10-15T22:48:43Z
2006-10-15T22:48:43Z
2006-10-15T22:48:43Z You must download this update to use the system.
2006-10-15T22:48:43Z
2006-10-15T22:48:43Z Download to your Applications folder?

I click on the "Download" box.

2006-10-15T22:48:43Z
2006-10-15T22:48:53Z WARNING: OSMessageBox: Automatic updating is not yet implemented for Linux.

Ha ha ha ha. Good one, Linden!

No, wait.

zoomj ipod

bbot — 2006-08-14T14:50:32-05:00

I just got my ipod working under linux. It was less than trivially easy.

First I had to get a firewire card. This was surprisingly easy. And then all I had to do was plug the ipod in and Linux automagically recognized it and mounted the file system.

Ha ha ha. Just kidding. This is what it did after I plugged it in.

[root@aluminum uqm-0.5.0]# dmesg -c
ieee1394: Node changed: 0-00:1023 -> 0-01:1023
ieee1394: Node changed: 0-01:1023 -> 0-00:1023
ieee1394: Node changed: 0-00:1023 -> 0-01:1023
ieee1394: Node changed: 0-01:1023 -> 0-00:1023
ieee1394: Error parsing configrom for node 0-00:1023
ieee1394: Error parsing configrom for node 0-01:1023
ieee1394: The root node is not cycle master capable; selecting a new root node and resetting...
ieee1394: Node resumed: ID:BUS[0-00:1023] GUID[000a270002012f83]
ieee1394: Node changed: 0-00:1023 -> 0-01:1023
scsi7 : SCSI emulation for IEEE-1394 SBP-2 Devices
ieee1394: sbp2: Logged into SBP-2 device
ieee1394: Node 0-00:1023: Max speed [S400] - Max payload [2048]
Vendor: Apple Model: iPod Rev: 1.30
Type: Direct-Access ANSI SCSI revision: 02
SCSI device sdc: 9780750 512-byte hdwr sectors (5008 MB)
sdc: test WP failed, assume Write Enabled
sdc: asking for cache data failed
sdc: assuming drive cache: write through
SCSI device sdc: 9780750 512-byte hdwr sectors (5008 MB)
sdc: test WP failed, assume Write Enabled
sdc: asking for cache data failed
sdc: assuming drive cache: write through
sdc: [mac] sdc1 sdc2 sdc3
Attached scsi removable disk sdc at scsi7, channel 0, id 0, lun 0
HFS+-fs warning: Filesystem was not cleanly unmounted, running fsck.hfsplus is recommended. mounting read-only.
SELinux: initialized (dev sdc3, type hfsplus), not configured for labeling

So I could see all the files, and listen to stuff, but I couldn't delete anything, or add any music.

But hey! What about fsck.hfsplus?

[root@aluminum uqm-0.5.0]# fsck.hfsplus /dev/sdc3
bash: fsck.hfsplus: command not found

Fuck!

So I get online and grab fsck.hfsplus. The problem is that it's a Darwin native app, so it's little-endian. Linux can't run little-endian code, so you have to apply a patch.

[bbot@aluminum uqm-0.5.0]$ cd /media/usbdisk1
[bbot@aluminum usbdisk1]$ ls
4chan hokuto~ mIRC
Archive HOWTO_hfsplus_files New Folder
diskdev_cmds-332.11.patch.bz2 HOWTO_hfsplus.htm Untitled.txt
diskdev_cmds-332.11.tar.gz imprints (another copy).html~ Untitled.txt~
diskdev_cmds.diff.gz imprints (copy).html~
hfsplus-patch-20020606.htm japan photos

[bbot@aluminum usbdisk1]$ tar zxf diskdev_cmds-332.11.tar.gz
[bbot@aluminum usbdisk1]$ bunzip2 -c diskdev_cmds-332.11.patch.bz2 | patch -p0

(a few hundred lines of patcher messages.)

[bbot@aluminum usbdisk1]$ cd diskdev_cmds-332.11
[bbot@aluminum diskdev_cmds-332.11]$ make -f Makefile.lnx

(a few hundred lines of complier messages.)

[bbot@aluminum diskdev_cmds-332.11]$ cp newfs_hfs.tproj/newfs_hfs /sbin/mkfs.hfsplus
cp: cannot create regular file `/sbin/mkfs.hfsplus': Permission denied
[bbot@aluminum diskdev_cmds-332.11]$ su
Password:
[root@aluminum diskdev_cmds-332.11]# cp newfs_hfs.tproj/newfs_hfs /sbin/mkfs.hfsplus
[root@aluminum diskdev_cmds-332.11]# cp fsck_hfs.tproj/fsck_hfs /sbin/fsck.hfsplus
[root@aluminum diskdev_cmds-332.11]# cd /sbin
[root@aluminum sbin]# ln -s mkfs.hfsplus mkfs.hfs
[root@aluminum sbin]# ln -s fsck.hfsplus fsck.hfs

Victory!

[root@aluminum sbin]# ./fsck.hfsplus /dev/sdc3
** /dev/sdc3
** Checking HFS Plus volume.
** Checking Extents Overflow file.
** Checking Catalog file.
Incorrect number of thread records
(4, 3)
** Checking Catalog hierarchy.
** Checking volume bitmap.
** Checking volume information.
Master Directory Block needs minor repair
** Repairing volume.
** Rechecking volume.
** Checking HFS Plus volume.
** Checking Extents Overflow file.
** Checking Catalog file.
Incorrect number of thread records
(4, 3)
** Checking Catalog hierarchy.
** Checking volume bitmap.
** Checking volume information.
** Repairing volume.
** Rechecking volume.
** Checking HFS Plus volume.
** Checking Extents Overflow file.
** Checking Catalog file.
Incorrect number of thread records
(4, 3)
** Checking Catalog hierarchy.
** Checking volume bitmap.
** Checking volume information.
** Repairing volume.
** Rechecking volume.
** Checking HFS Plus volume.
** Checking Extents Overflow file.
** Checking Catalog file.
Incorrect number of thread records
(4, 3)
** Checking Catalog hierarchy.
** Checking volume bitmap.
** Checking volume information.
** The volume iPod could not be repaired after 3 attempts.

Shit!

[root@aluminum sbin]# ./fsck.hfsplus -q /dev/sdc3
** /dev/sdc3
QUICKCHECK ONLY; FILESYSTEM CLEAN

And sure enough, dmesg wasn't showing the error message anymore.

Now I installed the linux equivilent of iTunes. Which worked beautifully!

Ha ha ha.

[root@aluminum uqm-0.5.0]# gtkpod
gtkpod: error while loading shared libraries: libgpod.so.0: cannot open shared object file: No such file or directory

So I had to export the enviromental varible, so gtkpod could find itself.

[root@aluminum uqm-0.5.0]# LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
[root@aluminum uqm-0.5.0]# export LD_LIBRARY_PATH
[root@aluminum uqm-0.5.0]# gtkpod

Ta dah. From there on out it actually worked.

The moral? Linux is only free if your time has no value.

But this would have been a lot harder a few years ago. My firewire card Just Worked, and the ipod initialized automatically.
But if I had a mac, I could have just plugged it in.