Site Issues (Mostly) Resolved
Between Pair Networks, my new hosting service, and myself, we seem to have determined the problems with slow loads and errors on comments. It turns out that Movable Type wants to rebuild the entire comment database every time it rebuilds pages (blog posts or comments) on a blog. We have a database of over 140,000 comments at CQ, and that created a database table that topped 160MB. Rebuilding that file every time created an overload on the server that caused their "reaper" programs to kill the processes before they ever had a chance to complete.
This was certainly the problem at my former hosting service. However, since they gave me no direction about the reason for the problem -- indeed, told me on more than one occasion that application issues were not their problem -- I had no idea how Movable Type handled rebuilds, and so I could not fix the problem. Pair spent more than an hour with me today, talking with their MT specialists and watching rebuilds live to determine the actual issue.
However, in order to fix this problem, I had to remove all of the comments from the system from December 2006 back. I do have them in a backup database, but it's simply not possible to carry that load forward. It interferes, unfortunately, with my ability to post and your ability to add more comments. I assume that threads more than four months old are probably dead anyway. I will have to maintain that kind of rolling limit in order to keep the blog running properly in the future.
The good news, however, is that I have eliminated Typekey and am relying on other anti-spam tools that run more in the background. Hopefully, they will cause a minimum of problems for commenters and we will be able to welcome more people into the CQ community. Thanks for all your patience, and hopefully we'll be back to normal or better by tomorrow.
Posted by Ed Morrissey on May 14, 2007 9:47 PM | Comments (22) | TrackBacks (0)
Comments (22)
Posted by unclesmrgol | May 14, 2007 10:59 PM
Now you just have to get rid of movable type. There's no reason why you can't have as many megabytes as you want without vacuuming killing you.
Posted by Bill Faith | May 14, 2007 11:13 PM
Test comment.
Posted by cathyf | May 14, 2007 11:31 PM
Make a note that if commenters get an "internal server error" that they need to wait several minutes to check to decide whether they really posted. I got the error, went back and refreshed the page, my posting wasn't there, so I submitted again and got the error again. This time I waited a bit longer before refreshing the page, and both copies of the comment had appeared.
Posted by Saul | May 14, 2007 11:33 PM
Long term planning might be beneficial in this case - you should consider WordPress (importing the database would likely cause quite the server load however).
Posted by marc | May 15, 2007 12:35 AM
Yeah... no more typekey tripe it's sec0nd only to it's creator, movabletype, on the suckage scale.
It's far past time for a switch to WordPress.
Posted by Kevin | May 15, 2007 12:41 AM
I used MT for awhile too, and the rebuilding the entire db every time I posted something got annoying real quick.
So when I switched hosts I took the opportunity to also switch to Wordpress. I have not been disappointed once. Sooooo much better than MT and your server will thank you.
Posted by Rose | May 15, 2007 12:44 AM
WOW! It is SCROLLING fabulously! I know you hate to lose a history rich in information, hopefully you'll find a way to save what is important to you, at least on back-up discs, if nothing else.
Lots of hard work you been pouring into this baby!
We do appreciate it.
Posted by TalkLeft | May 15, 2007 12:50 AM
Movable Type and its continual rebuild of the comments database was the reason TalkLeft slowed to a crawl last year. TypeKey was a dismal failure. I also tried deleting prior years' comments but that still didn't make it speedy enough. I ended up leaving Movable Type and making the site a Scoop site.
It cost me a lot of money, but the comments are now a breeze. The only downside is you can't edit comments in Scoop. You can only delete them.
Comment moderation is essential in this age. Movable Type just isn't built to handle the type of traffic and comments a blog like your's or mine gets.
I suspect it won't be long before you get so frustrated you look for another platform.
Posted by Rose | May 15, 2007 12:53 AM
Having slight problems typing in a post, but not like before! And then when I hit "POST", the rebuild fails, I have to go "back", and clean out the post, and hit "refresh", then the original post is there in the comments, ok...
Just occurred to me...wonder if my freebie "rose" cursor is causing a problem - I just remembered my son cleansed my specialty cursors from our computer, once, because of the "housemites that came with them... but my security system isn't catching anything, these days...
Posted by DubiousD | May 15, 2007 1:52 AM
Test.
Posted by Cindi | May 15, 2007 3:57 AM
Testing as well.
Posted by Patrick | May 15, 2007 5:49 AM
Thank you for ditching Typekey.
Posted by Patrick | May 15, 2007 5:52 AM
Thank you for ditching Typekey.However, now my first post in weeks is bring rejected
Posted by Patrick | May 15, 2007 5:53 AM
Thank you for ditching Typekey.However now my attempt to post is being rejected for "Too many comments have been submitted from you in a short period of time."
Posted by Captain Ed | May 15, 2007 6:04 AM
There is a 100-second timeout feature designed to frustrate spambots. It means you have to wait a little under two minutes to submit a subsequent comment.
Posted by mrlynn | May 15, 2007 8:14 AM
160 MB is tiny in today's world. I know nothing about web hosting software, but surely you can find software that can manage 160 GIGAbytes of back comments without hiccuping.
/Mr Lynn
Posted by mrlynn | May 15, 2007 8:18 AM
When I attempt to post, I get an 'Internal Server Error'. Then I use the Back button, and the post has registered, but it's still in the Comments box as well.
Something's screwy.
/Mr Lynn
Posted by PersonFromPorlock | May 15, 2007 8:43 AM
Why not use whatever hosting service 'The Volok Conspiracy' uses? It's fast, trouble free and lets them store articles back to at least 2004 and comments for the couple of years VC's been allowing them.
Posted by Cindy | May 15, 2007 10:11 AM
One oddball issue Capt. When I try to access the site from home, I get the pre-move archives. Nothing from May 14 forward is there. When I access from work I get the new site. I'm not sure if the problem is my end (I cleaned out my cookies etc thinking it might be a culprit) but that didn't fix it.
Does anyone have any hints for me?
Cindy
Posted by Al in St. Lou | May 15, 2007 10:32 AM
My Typekey account wasn't letting me comment here, anyway. Odd that it seemed to work at other sites. I was left wondering whether I'd been banned.
Posted by Al in St. Lou | May 15, 2007 10:35 AM
My Typekey account wasn't letting me comment here, anyway. Odd that it seemed to work at other sites. I was left wondering whether I'd been banned.
On the first try, I got an internal server error.
Posted by Cindy | May 15, 2007 11:19 AM
OK - brain fart from my last post - I should have said that my home computer only sees posts from May 10 not 14.....
Dang - I need more coffee.
Cindy