a timocracy of one

I’ve spent plenty of time kvetching about problems running my Typo blog on Dreamhost. I acknowledge, though, that despite some of DH’s other hiccups, the core problem is that running Rails, on Apache fcgi, on a shared host is a Bad Idea. And while I can handle the initial set-up of a slice/VPS I don’t want to deal with the maintenance. And I don’t want to pay an arm and a leg for a fully managed set-up, just to host small sites like blog.

So when my site started going down constantly, again, due to processing getting killed for being over the memory limit, most likely, I decided I would try out Dreamhost’s new “Private Server” offering, which is sort of a managed pseudo-VPS: cheaper than a true VPS, no root, they manage everything, but you do get virtualized/dedicated resources. Sounds perfect for what I need.

Unfortunately, there is a waiting list… and my site is down now! So I decide to email support and say, hey, I’ve sent some business their way over the years, my site is down now, any way to bump me to the front of the list. Well they did – but apparently side-stepping their usual processes. I show as still wait-listed in their web, panel, but all the sites go down on my account, all of a sudden, from the switch. It turns out they have moved me over to the PS, but the DNS didn’t get updated correspondingly for all my domains.

After a day or so of wrangling with their support queue, Jason got things fixed and it seems to be working. Now I just need to see how well it works for my needs. I’m going to just stick with the Apache fcgi, for now, because it works, I’m curious how the fcgi performs when not constantly being killed for exceeding the shared allowance of memory, and Dreamhost’s proxy set-up proxies you to a single mongrel by default, so I’d need to set-up up something like nginx to proxy to my mongrel clusters, and then keep all those up myself. Not that big a deal, but I am going to run with the out of the box experience a bit first, and see how little admin work I can do.

Some useful info on the DH wiki for git and on kernal.org’s guide to setting up git over http. The latter is especially helpful for debugging suggestions (like using curl to check your .netrc access permission). Using the info in these guides:

• Create a repo, or use an existing one
• Enable WebDav access on a DH directory, using their panel. Set-up the username password for the committers, and disable the password protection box for read-only anonymous access (might be easier to keep it on until you have everything set-up right)
• Create a .netrc with the auth info for this webdav
• Test your access using the curl command in the kernal.org guide (you may find it useful to have password protection enabled at this point,).
• Connect to the webdav share using the username and password.
• Copy over the .git folder contents into the root of the webdav share (this is key, because DH’s anonymous access won’t show . files)
• cd into the webdav share from your local machine and run “git-update-server-info”
• profit

You should now have access to http://yourserver/gitfoldername and http://gituser@yourserver/gitfoldername. For example: you can “git clone http://blerb.infosauce.org/git/” for read-only access to the blerb repo. Now, if I can only get gitweb working despite the limited access control imposed by DH webDAV.

Update: Changes the url for blerb to still work right after I changed it around while playing with gitweb. Also, you should use git://42squared.com/git/blerb if you actually want the real blerb repository.

Since I’ve been splitting up an app into an rails admin section and a separate camping front-end, I need to get camping working on the host to deploy. And that host, for this app, happens to be DreamHost, which means it’s gotta be fcgi. I’m not all the way done with that, but after a late night last night, I managed to get it running on fcgi locally. I’ll skip over the basics (for one thing, if you haven’t gotten this far yet, you are going to need to struggle through it to be familiar enough with what is happening to have a chance of finishing the process) until I have it all worked out smoothly, but I have to post a couple highlights, to spare some people with the same pain.

First off, no luck running off of edge. The only way I got it working consistently with mongrel, cgi (crucial for debugging and figuring out why things aren’t working), and fcgi, was to use the gem release (1.5.180), and patch that (keep reading). Once I had that running smoothly with the built-in server and then cgi, I got it working with fcgi. If you have the same problem I did, you’ll get an ”`camp_do’: undefined local variable or method `exc’” error. THis is because there is another problem, which is throwing an error, but there is a bug in the exception handling code. Manually fix that (change e on to exc in the exception catching, so that it is defined), and then you can move on to the next issue, since your exceptions will now actually get through to the log, or the screen, depending on how far you’ve made it, and you can use them for more debugging.

If you have my problem, you can track down the next error to the path variable not being populated right in the fastcgi.rb

    def camp_do(req)
        root, path, dir, app = "/" 
        if ENV['FORCE_ROOT'] and ENV['FORCE_ROOT'].to_i == 1
            path = req.env['SCRIPT_NAME']
        else
            root = req.env['SCRIPT_NAME']
            path = req.env['PATH_INFO']
        end

Wether or not I set ENV[‘FORCE_ROOT’] the wrong server variables are being set for camping to know the right path. This shows up as gsub on nil errors, or ”/dispatch.fcgi not found”, or some others, depending on exactly which way you are running things. This all comes back to not being able to use ScriptAlias, as in the wiki, due to only having .htaccess level access. And using RedirectRule and other mod_rewrite stuff that workins in .htaccess (I actually just stole that right from my rails .htaccess) populates things a little bit different.

My fix was to set “ENV[“FORCE_ROOT”]=1.to_s” in my dispatch.fcgi, and then change the corresponding “path =” line to match what my set-up was populating -req.env[‘REQUEST_URI’]. I tracked this down by getting things running well enough to get the default error dump to the browser, and then checking over that.

Now I need to duplicate this without using global gems (since I can’t upgrade them on the shared host), and the do more debugging on the actual host.

crontab

MAILTO=me@mydomain.com
# minute (0-59),
# |      hour (0-23),
# |      |       day of the month (1-31),
# |      |       |       month of the year (1-12),
# |      |       |       |       day of the week
# |      |       |       |       |
  */5    *       *       *       *   wget -q -O /dev/null http://mydomain.com >/dev/null
  @hourly /home/myuser/clear_timocracy_sessions.sh
  @daily /home/myuser/rotate_timocracy_logs.sh

Grabs the site every 5 minutes to keep the fcgi alive on Dreamhost, and calls scripts that clear the sessions and rotate the logs h3. clear_timocracy_sessions.sh

#!/bin/bash
/home/mysuser/mydomain/script/runner -e production "CGI::Session::ActiveRecordStore::Session.destroy_all( ['updated_at <?', 1.day.ago])" 

Clears out sessions older than a day. h3. rotate_timocracy_logs.sh

#!/bin/bash
cd /home/myuser/mydomain
/usr/sbin/logrotate -s log/logrotate.status config/logrotate.conf -f; touch public/dispatch.fcgi

Calls logrotate and then updates the timestamp on dispatch.fcgi, so Apache will create new worker processes h3. config/logrotate.conf

"log/*.log" {
  compress
  weekly
  delaycompress
  missingok
  notifempty
  rotate 52
  create
}

Archive weekly, keeping a years worth of logs. delaycompress – wait to gzip them, till the next week, both for ease of reading the most recent and so there is time for the fcgi threads to get killed off, since they are pointing at the old log.

After an email exchange with Dreamhost I have had it confirmed that their watcher processes monitor on a per user basis. This means that if you run each app under a separate user you are less likely to have problems with getting your fcgi processes killed.

Excerpt of my email exchange with Dreamhost

> Is it true that the processes are watched on a per user basis
> as in running each app under a different user name will
> mitigate problems?

Yep, that’s correct!

As you can no doubt see, this blog has return from its hiatus. As I’ve complained about before, running Typo on Dreamhost isn’t that dependable, so while I was busy wrapping up the sale of my condo and then moving to Tucson, I let it slide. But now that I am somewhat settled in, I’ve gotten around to the updating, tweaking, and praying required to get everything working again, and will probably be keeping a little closer eye on it.

I’ve turned on various spam protection measures (such as Akismet, disabling non-ajax commenting. moderation, etc), and am hoping that will alleviate some of the performance issues. If not I am thinking of moving onto a new system- looking for something that is less of a pain for me to maintain with this set-up, since I will be needing to get a job, and thus will have better things to do with my time than getting frustrated with Typo/Dreamhost-fcgi. I’ve been considering the newest kid on the block in the Rails blogging world SimpleLog, but maybe things will stay functional enough that I don’t have to go down that road and spend more time converting everything over.

Either way, I’ll be posting again here some, as I work on finally getting the Lost River Outfitter’s site updated with the new functionality that I promised them as a Christmas present ages ago, and then finally get around to playing with some of my ideas for InfoSauce 2.0. If I get a non-tech job (like my plan to work in a bike shop), I’ll probably be willing to spend more of my spare time developing, but either way I’ll try and post semi-regularly as I come across cool stuff.

As well, I’ll probably be working on getting the design updated with Stf’s work (after I feedback to her on the first round logo stuff), so I can get rid of the stupid placeholder header image, fixing the homepage nav in Safari, putting my resume up, and other house-keeping as I have time.

So I thought I’d hold onto Dreamhost as my primary host a while longer, but its driving me mad. It just won’t perform wortha snot, especially anything that depends on fcgi. Hence Trac and Typo (Rails is Apache fcgi on Dreamhost) are not exactly running up to snuff. But it’s not like anything has ever been going F1 speeds for me with them.

You want to know how I knew right away there was a comment posted on one of the entries? I have been giving the new Rails monitoring service, Heartbeat, a try, and I got a notification from them that my domain had changed from “200 OK” to timing out. Great, the load of serving up a couple page-views and handling a single comment pushed my Typo install to it’s limits.

I’m going to keep Dreamhost for storage, backup, and mirroring svn, since you get a lot for your dollar, but I can’t stand it as a primary domain any longer, even just for my blog and minor personal projects.

Dreamhost upgraded the RAM on my server yesterday. This meant it was down unexpectdly and I felt like that was teh last straw. BUT, I checked their special status page, and saw what was happening, and they did have it right back up around the time frame they said they would.

My sites perform MUCH better now. I’m still stuck with the vagaries of being on a shared host, with the (mt) gs set-up might be the solution for, but so far I haven’t had any problems. Maybe half my problem was just being stuck on an underequiped server. And after trying out the gs, I might be asking for a refund (they said they’d give me one, if I chose). I’m just too addicted to the featurefulness of DH. mt didn’t have SRV records (needed for the chat for GMail for Your Domain), email plus addressing (tim+randomstring@timocracy.com), anyway to run Trac, barring good (bad) old cgi, or mod\_dav\_svn (svn over Apache).

The solution for most of these would be too keep my DH account for those and use the Grid for things like Rails sites that need better performance. Well, why not just keep my DH only then, since my sites are performing fine now. I can always go back to mt later, after they add a Python container and SRV records (two things I know they are planning to implement). Thanks mt, for making it easy to cancel and get a refund, I’ll remember that when I need a little more umph, or if DH starts sucking again.

While Apache fcgi may suck, it’s better than that.

I’m getting really tired of Dreamhost’s non-stop problems. Hell even when something isn’t officialy broken my performance sucks. While I don’t expect to find mod_python so Trac will run nicely, for the Rails side of things (and all around, actually) a very nice looking new option sprung up: MediaTemple’s new Grid Server hosting. I’m seriously considering migrating everything over to their basic plan. Hell, I can keep my Dreamhost for random filestorage forever (well as long as one certain old client doesn’t get as tired of the performance as I am, and switch), given my referals (I might have to cut back to the basic plan).

I’m basically just waiting to hear back that Trac should run fine under .fcgi, which I expect it will, and to maybe read some reviews, since it JUST went live.

Oh, and the fact that DH just 10-foldified their disk space on plans, but are still DESPERATELY trying to get vendors to sell them a massive network storage solution doesn’t look good.

I just got Trac running on Dreamhost for InfoSauce (basically my) Open Source Projects to make it the home of the new Dynamic Scaffold Resource plugin for Rails

The Dreamhost Wiki entry on Trac made it a breeze, if only because of the link to an automated install script .

My only gripe would be that their section on Pretty URLs had a minor bug (for me).

DirectoryIndex wiki

should be

DirectoryIndex index.fcgi