metapundit.net

Smoke

Tue, 01 Jul 2008 13:57:23 PST

I thought this pic would make a good companion to the meta-wife's firemap. The area shown is north of where I live but the whole central valley has been a smoke-filled bowl for the last week and a half. It's actually clearing up pretty good in the last day or two but we skipped Mo-Band (a weekly concert in the park) last Thursday because the air was so bad...

Wow, just wow

Tue, 01 Jul 2008 13:58:25 PST

This article at the Weekly Standard is completely worth reading. For those who have followed the Charles Enderlin saga there won't be much new information but the attitudes of French journalists towards their sole non-negotiable professional obligation (to tell the truth) is still staggering. If you aren't aware of the controversy about the famous Al-Durra shooting at the beginning of the second Intifada, of course, this will make even more interesting reading...

You're killing me, man

Fri, 06 Jun 2008 13:13:09 PST

OK, you're killing me out there!

I'm turning into a crank, I know, but the quality of conversation around PHP development lately is really bugging me.

I ranted at length in the comments at the Sitepoint blog post about microbenchmarks and thought I got it out of my system.

No such luck - this morning in my feeds was a link to a blog post titled in_array is quite slow. DO NOT WANT!

I'm going to explain one more time what's bugging me and then start ignoring it...

First take a look at the really badly titled in_array post. Basically the author is repeatedly importing a large dataset from XML to a database and wants to eliminated duplicate records. The script starts running slowly (as in hours) and he discovers that the duplicate elimination logic consists of building an array of all the existing unique id's in the database and another array of the all the ids from the XML data. Then for each id in the list of new ids it searches the array of existing ids to see if the id already exists; inserting it only if it's new ...

With me so far? Notice how I cleverly didn't use the words "in_array". In fact, let's write some sample code to accomplish this algorithm without resorting to in_array.


foreach($new_ids as $new_id)
{
    $found = false;
    foreach($old_ids as $old_id)
    {
        if($old_id == $new_id)
        {  
            $found = true;
            break;
        }
    }
    if(!$found) insert_new_id($new_id);
}

So what's wrong with that code? Nothing, if my datasets stay small. The problem is that it's an O(n²) algorithm. This is Big O notation and completely worth reading about if you are or intend to be a programmer. Big O notation gives you a way to categorize the speed of algorithms. I've heard that O stands for "on the order of" but I don't see that on the wikipedia page. In any event, Big O analysis doesn't care about the actual speed of an algorithm, only about the way in which the number of operations varies as the dataset the algorithm is performed on varies. In this case it is pretty easy to see that if each of our lists has 10 items the inner loop will run up to 10x10 times (say if there were no matches between the two lists). If there are 1000 items in each list it will run 1000x1000 times. And to abstract this - if there are "n" items in each list the inner loop will run n² times - hence the Big O label. N squared is really bad because exponential growth doesn't scale the way we intuitively want things to scale. Instinctively I always assume algorithms are linear. If processing 1000 items in a list takes 1 second, processing 2000 items ought to take 2 seconds, right? That's true for O(n) algorithms only!

So back to the blog post - what's wrong with in_array() that makes it run so slowly? Nothing - in_array() is a search function - probably more sophisticated than my for loop code but essentially the same idea. Using in_array() in an inner loop gives you n-squared runtimes!

Now using the $old_ids list as a hash-table instead (values in the keys) lets you convert this back to a linear runtime. Replacing the for loop (or in_array) with


    if(!isset($old_ids[$new_id])) insert_new_id($new_id);

Makes the code run much faster. Down from hours to .8 seconds in this case. But notice that this has nothing to with in_array - instead it has to do with choices of data-structures and algorithms. It doesn't help that PHP's hashtable and list type are the same structure (that's not per se bad, as long as you are aware that there are two different such data structures)! We now have an O(n) algorithm because the speed of a hashtable lookup is independent of the number of items it contains. The whole "inner loop" becomes a single constant time operation that doesn't vary with the length of the input data. And of course a better blog title would be "Picking the right data structure" or perhaps even "Duh! Searching is slower than lookup!"

And here's my where my rant comes in. I'm not going to speculate about the causes (ease of use and ubiquitous deployment, in my book, plus the lingering awkwardness of the language that causes some of the solid developers to defect (eg: I still have Paul Bissex and Simon Willison filed under PHP in my RSS reader...)) but the quality of commentary in the PHP community is bothering me. Shallowness and bikeshedding abounds. The sitepoint post I mentioned at the beginning was pointing to yet another PHP microbenchmark discussion - should you use single quotes or double quotes around strings? Is while faster than foreach? To reference or not to reference when passing data structures around... All discussed un-ironically as "PHP best practices".

Stop it already people!

Here's some semi-constructive advice. Don't ever write about syntax unless you use the term newbie in the title. (The "for loop" for newbies is just fine). For everybody else - syntax is not programming! Saying you are a programmer because you "know PHP" (ie understand the syntax of the language) is like me saying I'm a painter because I can name all the colors. Especially don't pair discussions of syntax and speed - in fact don't talk about speed at all! Unless of of course you cite the rules for optimization:

Don't!
(for experts only) Don't yet

Or at least Knuth's aphorism ("premature optimization is the root of all evil"). In fact... I'm coining my own aphorism here (naming your statements makes them sound more official). Henceforth metapundit's law must be respected: don't optimize (or blog about optimization) unless you know who Knuth is. And if you've got some solid additions to my PHP feeds (is Harry Fuecks still writing?) I'd be glad to hear them.

The History of Christianity

Mon, 02 Jun 2008 11:58:33 PST

Thanks to nothing_to_say (and check out her new blog focusing on New Testament studies) I just finished reading Justo Gonzalez' The Story of Christianity. This is a two volume survey of Church history that covers the birth of the Church in Acts through through to the 20th century. I heartily recommend the book; it is an easy read, rarely polemical or particularly academic and the author makes the personalities and controversies of the Church come alive.

There's no way to comprehensively review a book that covers so much ground so I'll just record a few of the impressions I formed as I read.

One of the fun things about reading history is when you slot pieces of information into what you already know. I had this happen a lot. For exampe: Everybody knows more or less the history of Henry VIII, his wives and divorce, the birth of the Church of England and his two daughters who became Queens: Mary and Elizabeth. I was aware Mary was Catholic and Elizabeth was Protestant (got to love her Motto: "I see, and say nothing"). I never quite put the obvious pieces together however: Why was Mary Catholic? Why was Elizabeth Protestant? Personal belief probably had nothing to do with it - Mary was the child of Henry's marriage to Catherine of Aragorn. Henry tried to have his marriage to Catherine annulled and eventually split from the Pope over his (just) refusal. Elizabeth was the daughter of Henry's second wife Ann Boleyn. The Pope held that Elizabeth was illegitimate (and therefore a pretender to the throne) so of course she had to support the Church of England. Mary on the other hand was supported by the Catholic Church which rejected the annulment of of her mother's marriage. Of course she would be strongly Catholic... An obvious thing but an aha moment for me none the less.

Another thing I enjoy about reading history is when it causes me to rethink today in light of the past. One of the issues that Church history raises for me is the place of tradition. Chesterton describes a respect for tradition as democracy where the dead can vote! I'm coming from the radical protestant wing of Christianity (and if you'd read Gonzalez' book you'd know what that means) so I obviously have a different relationship to tradition than did Chesterton. It's worth thinking about though: Christians can be (I know I am sometimes) insular and complacent... Assuming that what I believe is obviously the truth and most Christians agree with me.

Tradition and history can give complacency a much needed smack on the head. I feel less certain of my theological positions when I discover that the vast majority of Christians throughout time would profoundly disagree with me. Take spiritual disciplines and exercises: how does it inform how I live my faith to discover the variety and consistency of spiritual discipline in the Church throughout history? Reading about early Church practices like weekly fast days makes think about the value of spiritual disciplines. Similarly - finding out that the early Church typically made applicants for baptism go through two years of instruction should inform how I view baptism.

I also had the opposite reaction when I located some beliefs of the Church in their historical context. I identify with the Internet Monk (who has long been one of my favorite blogger but recently has gotten even better: don't look now but I think he's trending anabaptist...) when he expresses affection for Catholicism because of it's non-trendy I-am-what-I-am nature. American Evangelicalism is ridiculously trendy and there's a certain attraction to a Church who just says "This is who we are. Deal with it."

That said - it's funny to read about the (relatively) recent status of some Catholic dogma that most bothers me. Transubstantiation (the doctrine that the elements of communion physically change to become the actual body and blood of Christ)? That didn't become dogma till the fourth Lateran council in 1215! A recent innovation... Papal infallibility (the idea that the Pope when speaking Ex Cathedra is free from error)? Declared by Pope Pius IX in 1870! This declaration gave force to his earlier Ex Cathedra pronouncement on the Immaculate Conception of Mary (ie: Mary was born free from original sin). In 1950 Papal infallibility allows Pope Pius XII to dogmatically declare the Assumption of Mary (the doctrine that Mary was taken up into heaven at the end of her life)...

The point is not that the ideas are recent innovations - but their establishment as Dogma (things you must believe to be Catholic) is a recent innovation. On the scale of Church history, at least, the Catholic Church has also had its evolutions.

That said - it is good for me to read a non-partisan history of Christianity. My ignorance of Church History is at least partially a Protestant ignorance. Protestant tellings of Church history (at a popular level) have tended to caricature: there was the early Church which obviously started right but gradually went off track and by the time Constantine shows up was completely screwed up until Martin Luther started straightening things out... In between (313-1517) there were a few good guys (I think we like St. Francis) but mostly the Church was a mess.

There's even a little bit of truth to that caricature: it is disheartening to read about the Church involved in wars and political intrigues - ecclesiastical figures with no interest in Christianity, division and confusion (three Popes at the same time!) and so on (and don't get cocky Protestants! All I have to say is 30 Years War!). Despite all the failures and fiascos the Church didn't disappear and the history of the less interesting but more devout leaders of the Church is worth reading for the reminder that there have been faithful through all times and circumstances. It is one Church catholic - though I sometimes forget - and especially for those of us who don't really tie our Christianity to history it is a worthy thing to be reminded and aware of tradition. I'm not sure that the dead get to vote, but they should at least get to nudge us from time to time.

Social Engineering at Google

Fri, 16 May 2008 10:35:05 PST

I just saw an awesome display of social engineering at Google in San Francisco (I'm waiting for the Google App Engine event to start). This might have been white hat - an authorized person just getting what they needed - but there's no way to tell.

A guy walked up to the receptionist desk, backpack in hand and asked "Hey are there any hotdesks open?"

The receptionist didn't seem to know what he meant so the "attacker" babbled on for a moment about meeting up with some people who might be moving to San Francisco google and who were "squatting" here. The receptionist offered to call somebody to check it out and the attacker quickly assured her that they were here unofficially so far and only X (didn't catch the name) knew they were here. "They said 4th floor though - are there any empty cubes back that way?" (pointing towards a secured entrance and walking towards it...)

"Yeah I think so..." responds the receptionist and buzzes him through.

As he's headed he calls back "Hey is there a microkitchen down here?" And the receptionist gives him instructions.

No ID. No name even given that I caught. And based on what knowledge did the attacker gain access to the googleplex? Just lingua franca - not all of it successful even! Say "hotdesk", "squatting", "microkitchen" and a mid level googlers name confidently enough and you either A) don't have to wait for your unofficial arrangement to be approved or B) score the google lifestyle (kitchen, cube, wifi) without actually working for them! Very nicely done...

Baypiggies, Post PYCON edition

Sun, 23 Mar 2008 01:21:56 PST

I went to baypiggies Thursday night for the first time in several months. The crowd was actually pretty small (some people were still at PyCON) but I had a really good time. JJ, Drew Perttula, and Wesley Chun presented on their PyCON experiences.

JJ has issued a flood of posts about his PyCON experiences. He said Martelli's talk on Callback Style Programming was the best (I think baypiggies is going to get an extended version of that talk sometime) but he also said the Raymond Hettinger talk on Python containers(lists, sets, dicts) was his favorite :)

Drew has a page up with links to the things he talked about. Based on his talk I have a couple of pieces of software I need to check out; particularly trestle (doctests for web apps) and supervisor (process monitoring)...

Wesley talked about a couple of things but ended up showing us some demo code that used the Google Spreadsheets API It looked really simple to read and write to the spreadsheet and especially paired with publically editable spreadsheets (or the new and awesome "make this spreadsheet a web form" functionality) I could see using this for some cool notification/cross machine storage capabilities. Wes also pitched his upcoming book a bit (NOT Core Python 2; that will be out in another 3 or 4 years he said). Wesley is writing a Django Book with the estimable Paul Bissex and Jeff Forcier. Django's getting a plethora of books all of a sudden and there was actually a bit of talk about Django at PyCON as driving some of the increase in interest Python. Apparently somebody asked how many people were programming in Python because of Django and got quite a bit of response.

This leads me to my own take away from the evening, however, which was to notice that there was a bit of Django-bashing as well. JJ contrasted Django's ORM unfavorably with SA (which is correct, but perhaps not as important as you might think.) I have hopes that as the admin app is decoupled from models and there's some movement to make a Django ORM->SA translation layer (see Michael Trier's blog) and have to admit that scanning the SA talk slides from PyCON makes me interested in playing with SA a bit.

More seriously, however, several people expressed great displeasure at the release state of Django. It's good that version 1.0 is coming (and check out the feature list: I'm really looking forwards to newforms-admin and model inheritance, but there are lots of other goodies there as well.) However! Nobody uses 0.96 which is a year old and the advice you'll get from everybody (including me) is to use trunk.

This does not tend to promote the impression that Django is stable, solid, and mature. I gave a little pitch for the impressive quality of the trunk code in my experience, but several people were fairly contemptuous about a project that never has releases. It also makes book writing hard (What are you supposed to say: use at least rXXXX or newer for the examples in the book but things may have changed already...) I've heard some of the arguments against 0.97 but I really think it needs to be done. Land one or two items that are done or almost done and roll it out, then head towards 2.3 or whatever we're calling the next release :)

Final and unrelated thought. Somebody was fumbling for the name of the "Consuming HTML" presenter at PyCON and a fellow behind me said "Ian Bickering, wasn't it?" I like Ian's work (<sarcasm>and I'm sure he's never heard that one before</sarcasm>) but I cracked up...

Eating Healthy on the Cheap

Sun, 16 Mar 2008 22:04:05 PST

If you know me well you know that I'm a bit of a foodie.

I don't have it real bad - I'm not a snob about much (except Coffee) and I don't really love cooking shows (Alton Brown excepted) nor do I have a gigantic collection of cooking books. Those disclaimers aside: I enjoy cooking and I enjoy exploring new techniques and flavours to expand my culinary palette.

There's a lot to be said for being a foodie. For me cooking is not just the drudgery necessary in order to eat: it's a hobby, a downright relaxing pleasure at the best of times and good practice even when food preparation is by necessity fast and familiar.

There are, however, some potential drawbacks to being a foodie. Health and expense are two prime concerns for me at least. It's not that preparing your own food is unhealthy (surely nothing can be as unhealthy as a diet of "whatever you can make from a box with a microwave") and cooking your own food is usually much cheaper than dining out. But enjoying food means... eating it! And somehow I have a tendency to prefer making rich savories rather than low calorie fare. Rich soups, sweet desserts... Pizza with pesto, walnuts, and gorgonzola! Steak well grilled, than drizzled with a Port butter reduction... Real tiramasu with brandy, espresso, marscapone cheese, cream, and ladyfingers! Cooking (as I have remarked when making something like alfredo sauce) is the art of adding calories until stuff tastes gooood....

Somehow that recitation of foods makes my mouth water (and my cooking itch need scratching) rather more than figuring out how to make healthier bread or lower calorie meals. Fortunately there is a second drawback to the foodie lifestyle that helps keep me down to earth - cost! That tiramasu had $30 worth of brandy, cream, coffee and marscapone in it! Meat is of course expensive and so are all the necessary accoutrements of exotic cooking. Creams, cheeses, wines, alcohols and vinegars. Spices (and of course we want fresh and whole instead of powdered and stale whenever possible)! All of it is expensive - hard on the wallet as well as the waistline.

Fortunately the solution lies in becoming a slightly more sophisticated foodie rather than giving up the enterprise. One of the things I look for in cooking resources are people who make "plain" food. Kevin Weeks (one of only two foodbloggers in my RSS Reader along with Tigers & Strawberries) often talks about "Peasant Food" and I really like this emphasis. Peasant food is working class food, stuff real people in various cultures have traditionally eaten. This doesn't necessarily make it familiar or even simple (for example Asian "peasant food" involves a whole spice/flavor vocabulary I'm barely aware of) but it does typically make it cheaper and healthier. Working class people haven't traditionally been able to frequently dine on meat with extra calories added - instead peasant fare frequently uses meat as a garnish or flavor but the bulk of the meal comes from grains or vegetables.

I've been working on the vegetable angle for a long time - I wasn't really a big vegetable eater as a child (unless corn and potatoes count) so one of my favorite things to do as foodie is wander the produce department and wonder what that

weird thing is and whether there is any way to make it taste good. Frequently there is - I've recently enjoyed foods that never saw my plate growing up - kale as a cooked green in soups, parsnips in latkes, squashes in vegetable mixes and pureed soups, red bell peppers in EVERYTHING... (No mom, I still don't like onions.)

I haven't worked much on grains until recently, however, and that's really a shame. Grains (and here I'm being expansive - I really mean to include legumes, rice, etc) tend to be rich in fiber, rich in vitamins, low in calories but high in volume, and cheap (at least some of them)! In the last year I've cooked particularly with Brown Rice (high in fiber, high in vitamins and less than $1/lb), lentils (tons of fiber and vitamins plus protein and a little more than $2/lb) and barley (also high in fiber and vitamins, also less than $1/lb.)

It's really easy to these grains to your cooking simply by supplementing existing recipes, particularly soups. Making soup with beef in it? Throw in a handful of pearl barley or brown rice - the soup will go farther, be healthier, and (I think) taste better. Add lentils to any potato soup - they'll cook down and be practically unnoticeable while substantially improving the nutritional value of the soup.

To a foodie, however, it's really worth it to investigate peasant cuisine rather than simply steal the ingredients. I've experimented with brown rice Paella and have been running through some Middle Eastern soups with lentils (and if you live in Modesto it's worth visiting the Middle East Market at the south-east corner of Coffee and Floyd if you need some inspiration or ingredients along these lines...) Food can be culinarily interesting, exotically good flavored but also healthy and inexpensive. You just have to look to some of the traditional staples in the dried foods aisle and stop obsessing about the 5 basic french sauces...

It's Magic

Fri, 22 Feb 2008 15:11:14 PST

Via phpdeveloper.org I saw this little blog post which notes that in an upcoming PHP 5.3 release a new constant '__DIR__' will be defined. Pretty obviously this is the directory equivalent of "__FILE__".

I know this doesn't seem like much of a feature but it actually makes me happy - a lot of my one-off PHP scripts start with:

define("__DIR__", dirname(__FILE__));

since I'm likely doing things like loading files from or looping over a relative directory... Not earthshaking, but does nicely reflect PHP's pragmatic impulses.

Is Google StreetView coming to Modesto?

Wed, 13 Feb 2008 12:38:57 PST

Coming back from dropping my daughter off at Kindermusik I spotted a strange looking periscope above traffic. As it got closer I could see a small blue sedan with a Google logo on the door. Mounted on top was a riser with perhaps 5 lenses attached radially. The closest resemblance I've been able to find online is the picture in this boingboing article. Is Google preparing to put Street-View footage of Modesto online? I might be in it if they do...

Modesto Symphony Nonsense

Thu, 29 Dec 2005 18:01:12 PST

I do not like reading fairy tales in my newspaper. Yet that, I suspect, is what has happened. Recently the Bee carried the news that the conductor of the Modesto Symphony is not renewing his contract. You can read the google cache of the article here (and my longstanding complaint about the Bee hiding their archives here). The article is headlined " Maestro's Finale - Music director chooses to leave symphony after this season".

The problem is this: I don't believe a word of it. I don't know anyone involved personally and I have only attended the Symphony's Performances a few times. I do know a little bit about the world of music, however, and this story absolutely does not make sense.

His contract expires in June and, at his choice, will not be renewed, orchestra President and Chief Executive officer Paul Jan Zdunek said Wednesday.

One (pronounced OH-nay) did not return phone messages left Wednesday at his Modesto home and office seeking comment. He also did not address the reasons for his departure in his statement, which came in an orchestra news release.

"I will always remember my six wonderful seasons with the MSO and I count it a privilege to have worked with an exceptional staff, committed board and talented orchestra," he said in the statement. "I am especially grateful for the love and support the community continues to give me and my family."

Somebody is not telling all they know. It is inconceivable to me that a professional talent like the Maestro has resigned for no reason at all. It is also inconceivable that an immensely popular conductor could be fired without any public outcry. It's too bad there is no local news gathering agency that could tell us what is really going on...

Django Blog Apps II

Fri, 07 Dec 2007 15:52:26 PST

Note: This is an update to my previous post looking for Django Blog Apps.

Since last time I looked at Django Blog applications a couple of heavyweight contenders have emerged. I've been meaning to point out Bruce Kroeze's Banjo Blog app for a while (currently not released but the source is available via Launchpad). Bruce has been among my RSS subscriptions for some time and also turned me on to Satchmo - Banjo looks like an attempt at a feature-full blogging app - skins, trackbacks, pinging, etc are all built in.

You may also be interested in BlogMaker. The folks at BlogCosm just released their blogging application and associated tools - I haven't run it yet but downloaded the tarball and poked around. The app looks pretty polished (uses Migrations to handle schema evolution, for example) and out of the box has support for comments, moderation, trackbacks, pings and a few other goodies. This may be the most polished Django blog application currently available.

It's also nice to see an "Other tools" section in their docs pointing out some of the other Django Blogging Apps available. Scott Lawton emailed me today (he must be making the rounds - I see links from Michael Trier, Simon Willison and the Daily Python URL popping up on my feed reader) and seems happy to work with other Django Blogging apps to come up with something approaching a WordPress competitor in terms of features and documentation. Not having quite had the time to port this blog to Django yet this is a goal I can enthusiastically get behind. Scott also politely pointed out a bug in the aging hand-rolled PHP I'm currently serving - obviously I need to quit commenting and start installing!

Wanderlust is gone

Thu, 06 Dec 2007 16:27:43 PST

Went by Wanderlust Coffee for an afternoon of caffeinated hacking and they were closed. A note on the door says they're out of business... Maybe not suprising?

DVCS for the Indie developer

Wed, 05 Dec 2007 12:25:07 PST

This is exactly right:

There are so many good reasons to try one of these new tools out. But I think the most important one comes down to this: just get it out of your head. Just commit the changes. Just start a local repository. Don�t create undue stress and open loops in your head about what, where, or when to import or commit something. Don�t start making copies of �index.html� as �index1.html�, �index2.html�, index1-older.html� �old/index.html�, �older/index.html� and hope that you�ll remember their relationships to each other in the future. Just do your work, commit the changes, get that stress out of your head. Share the changes when you�re ready.

It�s a much better way of working, even if it�s only for yourself.

I have blocks of code (especially inherited code) still not under revision control simply because there's a little mental hoop to get over in importing them into subversion. First is the difficulty in creating a repo from existing files; I always have to look this up:


# svn mkdir file:///root/svn-repository/etc \
         -m "Make a directory in the repository to correspond to /etc"
# cd /etc
# svn checkout file:///root/svn-repository/etc .
# svn add apache samba alsa X11 
# svn commit -m "Initial version of my config files"

From the subversion docs: how to do an "in-place" import

Ok, that's not really that bad - worse is the decision making process - should I just add this to my "everything repo"? Or is it possible that the customer will want a Trac instance at some point (or someone else will need svn access) and therefore I should make a new repo? (Oh, and since I rarely use the svnadmin command I always have to fumble around creating a new repo. Where's the repo root again? Should I do an http:// (if I need to share at some point) or file:// based repo? Does my new repo need a trunk and branches folder? All that adds up to enough mental friction that I frequently don't bother with revision control until I've been working on something for a while.

Using bazaar-ng all those questions are answered by a quick


#bzr init
#bzr add *

There's no reason not to go ahead and do it. And we haven't even gotten to the key benefit of DVCS - much easier branching/pulling/merging operations. I haven't played with git or mercury and only a little bit with darcs. If you've been telling yourself, however, that there's no point in looking at the new generation of distributed revision control systems because you work alone - pick one and try it out. Use it for a week. I bet you'll be suprised...

Warning!

Fri, 30 Nov 2007 00:29:54 PST

I'd like to talk about a blogger I've been reading lately, but I'm trying to figure out what my angle is.

This isn't a recommendation exactly - recently the blogger in question confessed to having been accused in conversation of running a "neofascist hate blog".

This is not an incredibly far fetched description of his opinions and topic selection. Kinda.

It's very strange to feel ... ambiguous ... about the writer in question since ambiguity is definitely not one of his flaws. I guess this is a warning (or at least a recommendation prefaced by a warning) about someone I've been reading lately.

Let me start by enumerating a few of his flaws. First he is verbose. Now I have been accused at times (particularly in technical writing) of going on and on. Let me assure you that while I have my flaws in this regard he is much worse! If brevity is the soul of wit he may be the stupidest blogger you'll ever read. In fact I view him as the alternate trek version of Instapundit - similarly spending 24 hours a day blogging but instead of posting 30 times a day he posts once a week. Please don't read him if you aren't willing to wade through lengthy essays!

Secondly he is impolitic. He swears. He expresses opinions in crass ways and expresses opinions that are ~~heresy against universalism~~ not often heard in polite society. Please don't read him if you are at all uncomfortable with considering opposing viewpoints. This is at a more profound level than whether one is a Democrat or Republican. I'm talking about accusing democracy of being responsible for the 20th century uptick in genocide. I'm talking about publicly noticing that Richard Dawkins is a deeply religious (albeit non-theistic) person who pretends (or really believes himself) to be against all religion. I mean the guy even caps on such foundational institutions as Penguin Books, NPR and Starbucks, for Peet's sake! Don't read him if you have blood pressure problems when people refuse to concede your obvious rightness in argumentation.

Thirdly he is often obscure. I consider myself to be well read in addition to reasonably conversant within my field of technical expertise. This guy is also conversant in issues of computer science but also has posted on economics, literature and philosophy in ways that go right over my head. Do you hate it when people make historical references you don't get? Use their own conceptual vocabulary? You're going to hate this guy!

Still reading huh. Well the blog in question is called Unqualified Reservations. And if you must investigate you might read his latest post on poetry. I laughed so hard I cried reading it to the metawife. But don't say I didn't warn you! Also if you want a deeper taste you might start in on his five part How Dawkins Got Pwned series. And if you're technically inclined you might like his utter condemnation of the University system and his follow up post on what's wrong with CS research. But don't come complaining to me about him...

Oh yeah, so why do I read this guy at all?

Perhaps you've heard a bit of the controversy about the upcoming Golden Compass movie? It's based on a book whose author, Philip Pullman, is allegedly evangelistically anti-religious, the trilogy of Books (the Golden Compass being the first) are supposedly anti-christian and Pullman has said some very unkind things about C.S. Lewis and The Chronicles of Narnia. It was with great interest than that I read the interview with Pullman containing tidbits like:

Interviewer: What sort of response to your books have you been perceiving from Christians? ...

Pullman: ... Christians at the other end, what you might call the thoughtful liberal end of the spectrum, have on the contrary been very welcoming. Many of my most interesting letters have been from, many of my most interesting conversations have been with Christians both Protestant and Catholic. They can see that I take these big questions seriously, and that the morality - the values that the book as a whole upholds and champions - is something on which we can all fully agree.

Interviewer: A number of commentators have argued that, while your books are critical of Christianity etc., they nevertheless reflect Christian virtues ...

Pullman: My answer to that would be that I was brought up in the Church of England, and whereas I'm an atheist, I'm certainly a Church of England atheist, and for the matter of that a 1662 Book of Common Prayer atheist...

Reading UR (particularly the Dawkins series) has made such exchanges not only intelligible to me, but highly hilarious. Your mileage may vary...

For Cheryl

Thu, 29 Nov 2007 18:25:26 PST

Blogging tip #143: When people at your Church are complaining about the lack of content on your blog - you've got a problem.

It hasn't been a lack of topics that's kept me from blogging lately - I have list! But the list has started to feel overwhelming lately: I have six or seven different posts mentally composed in my head just waiting for the three hours of writing required to get them all out of my head and onto the screen.

But who has three hours? I think I'm flushing the list and clearing out a little mental space... Ahhhh, I feel better already!

Now - what's been going on lately?

At the In-Law Thanksgiving evening we dropped in on we got an abbreviated version of last years What are You Reading/Book Recommendations talk. The recommendations (that I can remember) are as follows:

The MetaWife is reading Charlemagne. It's a bit of a polemical biography by Derek Wilson arguing that the 8th century Frankish King is basically responsible for the Concept of Europe. I guess we can blame the EU on Charlemagne! Anyways this has been academic (it's taken a couple of checkouts from the library to get through) but interesting (she's still at it).

Kristen is reading Neil Stephenson's Snow Crash and finding it tough going. Here I must confess myself biased (I think she's reading ~~my copy~~ the copy I bought for my wife) - Snow Crash rocks! The cyberpunk side is among the best I've read (read this and Gibson's Neuromancer and The Matrix seems a lot less original) and the meatspace r/l side is a great action yarn. The political anarcho/libertarian u/dystopia is drawn with a Heinleinian eye for quirky details. Add to this a dose of mind-bending neurolinguistics (brief summary: the tower of babel story and Sumerian mythology detail real history in which humanity lost the original hardwired proto-language/verbal virus due to the work of a primitive neuro-linguistic hacker (the Sumerian god Enki). No really.) Keep at it Kristen! (Why do I see A Girl of the Limberlost on your current reading list?)

Delbert is reading another Paul Theroux - last year it was Dark Star Safari (Theroux's account of bumming rides across Africa). This year we have The Old Patagonian Express. Who knew you could take the train from Chicago to Argentina? I love that there are still people with a personal sense of adventure - to go out the door and head someplace taking whatever comes along the way...

I've been reading more biographies lately - Bill Donovan, Francis Drake, and most recently Richard Brookhiser's Alexander Hamilton. Brookhiser writes a pro-Federalist, anti-Republican account of Hamilton and his well known political struggles with Jefferson, etc. Among the tidbits I've discovered while reading the biography (and this is why I value biography: the random connections that flesh out history): that the execrable Aaron Burr is Jonathan Edwards grandson and that Hamilton (who famously fired into the air in his fatal duel with Burr) had advised his 19 year old son Philip to do so as well in his own fatal duel a year previously!

For the rest - Caroline was reading the American Century series by the prolific Gilbert Morris. I may have missed a review or two - Cheryl and I got into a conversation about founding fathers (she having recently read a John Quincy Adams biography) and I lost track of the conversation. Email me if I've missed anybody with a recommendation.

Update: Oh, and I did have one other recommendation to make. Mark Helprin is one of the few recent authors I've read whose powers of prose are arresting. The MetaWife and I recently read A Soldier of the Great War and had to constantly read passages to each other. I must confess to not liking Winter's Tale but I am reading and enjoying a short story anthology of his The Pacific and Other Stories that has also proved riveting. Very strongly recommended.

Python Module of the Week

Tue, 16 Oct 2007 22:19:24 PST

Doug Hellman's Python Module of the Week series has been uncanny at frequently highlighting modules just as I'm learning about them. As a relative Python newbie I have a lot to learn about the standard library. I've got the excellent Python in a Nutshell which has a good tour of the standard library but it's nice to have a module pop up in my rss feeds on a regular basis to keep me learning.

Recently Doug covered difflib which I just been exploring as part of a regression testing suite I put together for a legacy web app I maintain which has no other automated testing. I'd run into a roadblock using the junklines option of the various diff engines and left on comment on Doug's PyMOTW entry. Not only did he explain my conceptual error but he emailed me when he put up a response to make sure I saw it!

One of the things that has struck me as I've moved from primarily PHP work and interests to primarily Python interests (if not yet work) is the ways the communities are different. I anticipated that the Python community would be smarter and more Computer Science-y, for lack of a better term. This has definitely been true - the PHP community doesn't have anyone of the caliber of Guido - and even "lesser" lights in the Python firmament are impressive. Alex Martelli is one of the smarter people I've met, for instance, and I doubt there are many people in the PHP community with his breadth of research and applied software experience. I was a bit worried, however, that the Python community has a reputation for being a little a little ... stiff. After all, there should be one-- and preferably only one -- obvious way to do things! (Admittedly, this impression of stiffness may have been garnered from ROR bloggers.)

So far, at least, I haven't found this to be true at all. Doug is a great example - not only is he adding knowledge to the community but he's going out of his way to be helpful and engaging to newbies like me.

Chinese Food and a Great Food Blog

Thu, 11 Oct 2007 21:19:41 PST

I've been making a lot of stir-fry lately and I was trying to decide if I should add a wok to my Christmas list. I've got one nice commercial grade Look skillet that I do most of my stir-frying in but it's pretty shallow and doesn't hold as much as I'd like to make. Now a wok is a pretty cheap pan as cookware goes but as soon as I started looking I discovered there are options to consider. Do I want carbon steel? Cast iron? Light or heavy? Hand hammered? And I have a flat top electric range - that means I have to get a flat bottomed wok, and will I have enough BTU's to really stir fry well anyway?

I decided to hold off on buying a new pan when I saw the wikipedia page about woks note that some people stir fry in cast iron skillets when limited by the BTU's of their stove - cast iron skillets tend to be very heavy and retain heat well - if you give them enough time even on a weaker stove you get them as hot as you need. I love my cast iron (and have a big 14 incher already) so now I'm eager to try it out. I hadn't thought of using my cast iron pans this way; I usually really work to keep the heat low in my cast iron skillets to keep their non-stick properties. With stir-fry you're really frying things so this may not be a problem - I guess I'll find out.

While all this research was going on I stumbled across a gem of a food blog - Barbara Fisher's blog Tigers & Strawberries has nearly 100 posts tagged "chinese recipes" and I found her when Google directed me to this fantastic post on stir fry technique. I can tell right now that she's going to help expand my range of Asian cuisine and techniques - Pan Fried Noodle Pancakes? Cold Sesame-Peanut Noodles? Or some Thai food - Panang Curry Beef? I've got some recipes to try...

Form Saver Firefox Extension

Tue, 09 Oct 2007 09:54:51 PST

I recently discovered the Form Saver extension for Firefox. This plugin harvests the current state of a form and saves it as a Javascript bookmark. This proves invaluable when you have a moderately complex form to test and don't want to have to repeatedly fill out test data. In my case I had several different data sets I wanted to enter and using Form Saver I made a series of Bookmarks in my toolbar folder. This meant entering the testing data on repeated runs just meant clicking a button. Very worthwhile for anybody doing web development!

Blogging Backlog

Sat, 29 Sep 2007 11:14:23 PST

I'm seriously behind on my blogging - I've got half a dozen things jotted down with notes to blog about them sometime... I've been delaying mentioning the Django Sprint until the main bug I worked on is closed (ahem... )

Ah well - some random snippets from the last week or two:

The last thing I need to do to get a final inspection on my addition is build a step from the new back door down to the new patio - a distance of 16 inches. I drew up some rough plans for a little deck step with 45 degree wings that would look nice and ran it by my dad as I was borrowing his truck. No go, he says - code requires (and I should have looked at my plans before messing around) a 36" landing at the full 16" height outside the door, than 3 steps with 1 foot tread and at least one handrail. All this to span the 16 inches.

I had to redraw my plans and significantly upgrade my budget - here's the rendering of the frame for the first level of the landing + steps:

Yes I know Sketchup would be a more appropriate tool than POVRay - but SketchUP doesn't run on Linux and POV-Ray doesn't require that I learn some weird UI like every other 3d modeling/2d CAD program out there.

Speaking of construction - I went to the annual Customer Appreciation banquet at American Lumber the other day. We've been going to the dinner for several years - it's real guys night out with a dinner consisting of meat and your choice of several different sides of meat. Nothing like Tri-Tip with a side of BBQ chicken, pulled pork, and smoked salmon... They have served alcohol at least the last couple of years that I remember but I've never been tempted too much by the Bud/Coors on tap selection. Contractors' tastes must be going upscale, however, since this year the choices were Fat Tire Belgian Ale and a Blue Moon Wheat Beer. Good stuff - just too bad that I continue my record of futility in failing to win even a T-Shirt in the post-dinner raffle...

I think the battery on my cordless Skil-Saw is about charged up so I'll get back to it. Coming soon - articles on using Web Service API's from SmugMug (yea!), UPS (hmmm. A little clunky) and producing and consuming QuickBooks XML (yikes!)

Just because he asked so nicely

Thu, 20 Sep 2007 23:55:14 PST

If you're looking for a Common Lisp tutorial, you should definitely read Peter Seibel's book. I've read bits of it and working through it is on my list of things to do (and Paul Graham's On Lisp and Norvig's Paradigms of AI Programming. At which point I will probably ascend to a higher plane of being or something...) Anyway - apparently because Lisp is such an ~~old~~mature language, Google hasn't always pointed at the best resources so Seibel is asking for a little love by way a helpful Googlebombing...

Baypiggies Again

Fri, 14 Sep 2007 00:11:21 PST

Just got back from the Bay Area Python Interest Group meeting at Google. The presenter tonight was from SnapLogic and went over their open source data transformation application. Sounded mostly outside my problem domain but the presentation was interesting...

I also got to harass JJ Behrens about his xpath macros for Python idea. Even as just a new way of handling monkey-patching, this is interesting. As a full on recursive macro expansion facility for Python... Interesting!

I'm still open to taking anybody interested in Python or programming in general to Baypiggies. It's the second Thursday of each month - email me if you're interested. Did I mention Google provided free Ben 'n Jerry's tonight?

Coffee in Modesto

Sat, 25 Aug 2007 18:18:16 PST

The Meta-wife and I went out for lunch and shopping the other day (finally using a gift certificate to Yesterday's Books we got as a Christmas gift... Thanks Dad!) After the book buying we went to Wanderlust Coffee for lunch.

View Larger Map

Wanderlust was excellent with a nice selection of teas, good coffee (iced Americano in my case) and excellent (if slightly pricey) crepes. We shared a Smoked Salmon, Cream, and Capers Crepe that came with a small salad of tossed greens and a drizzle of honey/balsamic vinaigrette (I think) that was very tasty (approx ~$7). Followed up with an apple caramel dessert crepe (~$4) and made for a very satisfying lunch.

Unfortunately it looks like the original very talented barrista Tony Serrano is on the outs with the owners and no longer associated with Wanderlust. I'm with Chris Murphy - Serrano should re-open the much missed J Street Cafe! For a while when J Street Cafe closed there wasn't any good downtown independent Coffee - now with Queen Bean, the re-opened Claytons, Wanderlust, and the new Semplicita Coffee (which I haven't even been to yet) - we've got an embarrassment of riches.

Update: Hmmm. The Wanderlust Drama heats up! Note also the comment by what looks like a newly registered account at ModestoFamous... Reads like sock puppetry to me!

Update: Closed!

Recipes

Wed, 29 Aug 2007 00:07:02 PST

I was just telling Kristen the other night that I like it when she food blogs. Since then I've made a couple of oddball things that turned out good so I'm dropping pointers to the recipes here. No pictures, unfortunately (ahem. Crashd - where is my camera?)

First I'm trying to eat brown rice so I browsed various recipes at epicurious (as per Kristen's suggestion) and found some ideas. I've tried variations on both recipes and both turned out really well - it's just taken a little experimentation to get my brown rice cooked just right (60 minutes! to get slightly al dente). Frying it briefly first yields nicely separated grains and cooking it in chicken stock gives it a lot of flavor. Adding the spinach (and fresh basil and sauteed red bell peppers last time) really makes it an awesome meal - relatively healthy, very flavorful, colorful, and easy (if a bit time consuming - you can't decide to cook brown rice at the last minute).

I'd also stumbled across 3 African recipes the other day and filed them away for for future reference. Tonight I made the Peanut Butter Soup and the Nshima (which is just a mush made by cooking cream of wheat down to a thick texture.) I was worried initially because the soup was a little thin, but chunks of the Nshima are just the thing to thicken it up. Excellent flavor - slightly sweet (due to the peanut butter) but nutty and spicy also. I did a few tweaks (no onion, a few cloves of garlic instead) and could probably make it a little spicier. Now I'll have to try the Nigerian Spinach Stew from the same page - the other two recipes are definitely winners!

Django application mini-review

Sun, 26 Aug 2007 23:55:20 PST

Update: check out my followup post for some better options.

Ok, there's a definite need for an official django-application repository, ideally with the ability to sort and filter by activity level and user ratings/reviews.

Consider - I want to find a blog application that runs under django. Now it's true that this is a trivial application I could whip out myself, but why bother re-inventing the wheel? Let's look for a django blog application. I've got a few ideas for features I'd like in a blog package (markdown or richtext editor, tagging, maybe ping-o-matic integration) but in general I'll just take something that works...

First stop is the ~~cheeseshop~~ python package index or pypi as we're supposed to call it now. This is the central repository for python packages that will be easily_installed. Searching for django blog yields approximately 25 results with a matching score of 3 or better. None, however, appear to be django applications - there are turbogears blogs and python blogging tools of various sorts, but no django applications.

Ok, well packaging a single django application to be easy_installed in site-packages might well be overkill - maybe django applications tend not to be packaged. I'd be fine with pulling straght from SVN - lets look at some traditional open source software repositories.

Sourceforge is a well established site for hosting open-source projects and has good tools for checking out the freshness of projects - dates, activity rankings, bug lists, and so forth. Searching for django blog brings up over 600 results. Sorted by relevance to the search term, even on the first page most of the results are not django blog applications (other django projects, php blogging applications, etc). The only real hit on the first page is blango which has low traffic (svn revision 8, no bugs, patches, messages, etc) and has svn commit messages in french.

Ah - there's the advanced search link. Ok, requiring both django and blog returns ... 1 hit. The aforementioned blango. Ok, so django people aren't using sourceforge. Oh, that's right, all the cool kids are using Google Code now.

Now this is more like it! Searching for django blog brings up 29 hits and on the first page most of the projects are django blog applications! Very nice - now how to pick the one I want... There's no way to sort/filter by activity level. No way to tell how recently the project was created and if it's been kept up-to-date (this being pretty important with a moving target like django - a app based on 0.9 isn't going to be that useful to me when my sites are surfing on the latest svn release of django). Advanced searching features (I believe Google knows something about search?) would be really useful here.

Be that as it may, it looks like pages 2 and 3 of results aren't actually that useful - lots of "this is my whole website" type of releases and less reusable packages. That leaves me with 7 packages to check out. So:

waggly-blog has two project owners, a couple of wiki pages (both recently edited) and is on revision 21. A quick look at the web based svn repo reveals a feeds.py and a templatetags directory. This sounds promising.
djog has one owner, no wiki pages, two bug reports (one dated November of 2006 - parenthetically it's really annoying not to be able to tell how fresh the project is by going to the homepage) and is at revision 4. Ehh...
becca has one owner but has a busier front page - a link to the blog of the developer and even a zipped download package. It uses django-tagging to support tags, textile for editing (another text format language like markdown, restructured text, etc) and saves the rendered html when the text changes instead of re-rendering on every page load. Ooh the repo is on revision 26 and the developers site is django based and exposes databrowse (which I hadn't played around with yet)! And a new contender emerges from the pack!
mauvaiston has three owners! The wiki has one page with default text, however, and the repos is at revision 2. Ooh and the trunk is empty. Google code says it's hiding the empty projects and I guess this is "non-empty" because they created the default wiki page. Shouldn't projects with no actual code be excluded as well? Moving on...
jblogexample's front page says simply "Self education project". This is probably enough to keep me looking on but checking quickly - no wiki pages and svn revision 5. No harm in releasing your learning projects, but I'm looking for something hopefully already pretty polished...
maplyeblog has one owner, no wiki pages, and is on revision 12. Trunk has a "tagging" application as well as the "blog" app and a couple of different settings files for different setups. Might be worth a look...
super-blog says it's a multi-user blog system which sounds promising. It has a link to the developers blog but the site doesn't come up. The svn revision is only at 2 and again there is no way from Google's web interface to tell when the project was last modified. I'll pass...

That about wraps up the Google Code results. Looks like I'll be checking out becca with a fall back to waggly-blog if I run into show stopping bugs. Obviously, however, there needs to be a one-stop shop to do this sort of research in a more convenient fashion... It occurs to me I could write a django application rating site - a quick search to see if somebody's had the same idea finds somebody is at least thinking along the same lines. I realise there's discussion around having an official django application repository - but in the meantime while sorting out issues around installation and hosting and so on, could we at least get a central site at which to search and rate django apps?

Leningrad Cowboys

Fri, 10 Aug 2007 23:29:14 PST

I can't even really describe how surreal the Leningrad Cowboys are. Any band self described as influenced primarily by "polka and progressive rock" (I grin just typing that) has got to be weird. Add in the fact that this Finnish rock band specializes in covers of American Classic Rock while playing accordions, really big Balalaikas, and guitars shaped like tractors ... Did I mention they're frequently accompanied by the Soviet Red Army Choir - a 160 member male ensemble of brass and vocalists, all clad in Soviet Uniforms? Oh and dancers, traditional Russian dancers... You really owe it to yourself to check out this (ZZ Top's "Gimme All Your Lovin" plus Dasvidanya & a smidgen of the Hallelujah Chorus). Or maybe this - Sweet Home Alabama with a Russian accent!

Somehow this all makes me think of Neal Stephenson...

Mt Whitney

Wed, 08 Aug 2007 22:01:13 PST

Well that was the worst experience of my life. I think I'd like to do it again!

A couple of weeks ago I hiked Mt. Whitney. One of my friends organized the expedition and ten of us went. Now many years ago I did a lot of hiking in the Sierras. My brother and I went to a "Hike Camp" up at Peaceful Pines for a while. This is in the Clark's Fork area off of 108 and is at about 7000 feet in elevation and within an hour of lots of interesting hikes - the Dardanelles, Lightning Mountain, Sonora Peak, Leavitt, Teapot. Most of those are between 6 and 15 miles and none of them get higher than a little over 10,000. I won't say I used to scamper up the mountains, but I did actually enjoy a hike a day for a long weekend - perhaps 50 miles in 4 hikes.

Like I said, it's been a while since I hiked much so I knew I'd be gutting out Mt. Whitney. Whitney is a 22 mile hike with some steep grades, but the real killer is the elevation. The top of Whitney is just under 14,500 feet - the highest point in the ~~continental~~contiguous US.

It was worse than I thought. Much worse in fact: on Friday my group met out in Oakdale at 5:30AM and drove through Yosemite south to Lone Pine. We met up with the other group (who had already slept the night at elevation) about 12:23 PM and headed up the Mountain to our campsite. We only hiked in 3.8 miles to Outpost Camp but I was already experiencing altitude sickness at 10,000 feet (mostly nausea). Fortunately the hike was short - after I laid down for about an hour I felt better and ready for supper - TriTip sandwiches, garlic mashed potatoes and apple pies afterwards. I was (unfortunately) still feeling a little to ill to enjoy the Sierra Nevada I'd packed in (all that effort for a six-pack everybody but me enjoyed!) but the hot food did stabilize my stomach.

We slept the night under an intense canopy of stars. There were so many stars the sky shimmered. When I mentioned in the morning that I almost woke somebody up to see if I was hallucinating one of the other members of the party (a Canadian who'd seen them before) also said he thought he could see a touch of the aurora borealis. Whatever it was, the night sky was beautiful!

It didn't take long in the morning to forget all about the beauty of the night. By the time I got up to Trail Camp I was feeling sick again. After resting a while by the Lake we pressed on - at this point I was only hiking with 4 guys from my group. Ascending up the mountain from Trail Camp is a climb of 1,700 feet and (allegedly) 99 switchbacks. I stopped counting after about 30 and focused more on whether I was going to throw up or not.

I've really never felt so miserable for such an extended period of time. It took me a couple of hours to get to Trail Crest at the top of the switchbacks and I'm pretty sure I would have given up if my friend Justin hadn't stayed back with me. I was walking like videos of Everest climbers I've seen - Each step taking an interminable time and covering only 8 inches or so. I couldn't eat or really drink due to the nausea and abandoned my pack at about switchback 80 - taking a single liter of water on with me and leaving a couple quarts of sports drink behind.

I had a lot of time to think. Cyrus and I have had some conversations about the role of suffering in the Christian life lately and the conversations ran through my head - I told myself that I felt miserable but that was OK. I could be miserable for a while and it wasn't the end of the world - in the meantime I just needed to keep hiking.

So I did. When I got up to the trail crest I rested a bit and went on under the false assumption that we were "almost there". We were, and we weren't - Trail Crest is only 1.9 miles from the summit but it took almost forever. Strangely, however, my altitude sickness abated. I didn't feel sleepy or dizzy any more and had no temptation to throw up unless I tried to eat or drink. I hiked the long gradual climb around Trail Crest to the summit a little better (Justin didn't have to wait for me anymore, at least) and finally got to the summit about 2:00PM.

Coming down was awesome. Downhill was easier and every step brought more air to my lungs. There's a reason everybody headed downhill had a silly grin on their faces. I even felt well enough to enjoy the view (panoramic, but not especially beautiful - that high it's like a lunar landscape with no vegetation or trees to relieve the tedium of rocks and dirt). By the time we got back down to the campsite I was thirsty enough to drink the Coke I'd left in the creek. I still couldn't eat (since breakfast I'd only had about a 1/4 cup of gorp) but I was perking up. We packed up our trash, picked up the bedrolls and headed down at a teeth-rattling clip. We just barely beat the darkness back to the trailhead.

Oh my. That was definitely the hardest hiking experience I've had. It's challenged me to get into better shape and I've been biking a bit since then in an effort to do so. In addition to being in better shape and weighing less, however, I think it would definitely be worthwhile to spend some time at altitude before hiking. I've never been really altitude sick before and it's not an experience I want to repeat... But summitting Whitney... Yeah I may have to try again.

Update: My brother thinks he needs some hiking props. Both he and my brother-in-law have been doing a lot of hiking lately - the week before our Whitney trip they did 46 miles together - one trip up Half Dome (20 miles) and one there-and-back-again overnight into the Emigrant wilderness area of 26 miles round trip. They both think they're in much better shape than me. They're both right.

They actually got a head start on the rest of the group on day two of about half an hour. Chad (who is a hiking monster) caught up to them and they were the first three to the top - Cyrus took about 3.5 hours to hike the approximately 6 miles and 5,000 vertical feet from outpost up to the top (and I, by comparison, took about 7 hours to get there). Interestingly my brother-in-law got a little light headed on the Ridge Crest trail above the switchbacks and lost the other two - it took him approximately 2 hours longer to hike the last 1.9 miles than it took the two leaders... Of course he still beat me by 2-3 hours!

Python Performance

Wed, 08 Aug 2007 22:48:53 PST

I've been emailing back and forth with a friend who is also learning Python. He is a physicist, rather than a programmer, and is focusing on the data crunching abilities of Python in ways that I haven't explored.

One of the things he was trying to do is load some data representing a 2d array from binary file and plot the points. He'd figured out the loading (via the struct module) and the plotting but was noticing that he had a few bogus values in the array that he needed to replace with zero.

We wrote the naive implementation together - his initial data structure for a 2d array was lists of lists and my first thought was to loop over the outer list and apply a filtering function to each of the inner lists like this:

def not70(x):
    if x == 70:
        return 0
    else:
        return x
    
def replaceList(a):
    """loop through 2d list with for loop, using map to find and replace
    values in lines of array"""
    for index, line in enumerate(a):
        a[index] = map(not70, line)

This worked, but it took a lot of time for his very large data sets (something like 10-15 seconds).

I was inspired to check out how we could improve this. My first thought was to switch from using lists to using arrays. Python has a built in array type that holds homogeneous base types (as opposed to lists which can hold objects of any type heterogeneously). After one blunder (filtering an empty list is faster than filtering a big one, who knew) I found that using the same algorithm on a list of arrays was actually slower than on a list of lists!

Hmm. Well I know that the built in arrays have a rep for being slow, hence the presence of NumPy. My box (Kubuntu 6.06) is still on Python 2.4 and didn't have NumPy installed by default so I started out with the Numeric module. Now, interestingly enough, using the same algorithm was even slower yet (those built in lists sure are optimized). I knew we could do better, though, if we could express our filtering as Numeric array operation. After a little bit of thought (the Numeric api was not always obvious to me) I came up with:

def letsUseALittleMath(a):
    b = Numeric.not_equal(a, 70)
    a = a * b

The first line results in an equally sized array "b" filled with 1s where the condition is met (an entry is not equal to 70, our supposed "bogus" value for illustration sake) and 0s where the corresponding position had our target value. Simple multiplication in Numeric is implemented so that in the resulting array c, c[i][j] = a[i][j] * b[i][j]. And of course multiplying a cell by zero yields zero and multiplying a cell by 1 yields the original value.

This was indeed a lot faster - in the tests I'm running with the timeit module, using replaceList above on a large list of lists takes 28 seconds for 500 iterations, with 39 seconds and 59 seconds respectively for the same algorithm on the presumably more efficient data structures of a list of built in arrays and a Numeric 2d array. Using the built-in array operations only took 3.5 seconds for 500 iterations.

That was a lot better, but Justin wanted to try Numpy. I `sudo easy_install numpy`ed and once gcc was finished chugging ran his script. He re-implemented the filtering function using a logical condition in an array index as Numpy allows (very cool - how do I support that in my own classes?) to to come up with the one liner filtering function:

def fastway(a):
	a[a==70]=0

Using Numpy and the built in syntax for replacing values like that resulted in a 3x speedup over the algorithm I implemented in Numeric. Here's the file that right now is yielding:

Time using all lists: 30.540692091
Time using lists of arrays: 33.4445209503
Time using Numpy arrays and array multiplication: 3.20908308029
Time using fastway: 1.08567285538

This seems like relatively decent performance (I haven't implemented in any other languages for a baseline) and is especially nice that it's relatively easy to optimize: we're both Python newbies but I only have about an hour of hacking and reading invested in improving the performance of the operation. It's also possible that there's a more straightforward/obviously faster way of doing this - I wondered about filtering the values before going to the 2d array - but this was a fun learning experience. Lessons learned - stay away from looping over large datasets and don't bother with built-in arrays (go straight to Numpy).

Annoyances

Sat, 14 Jul 2007 12:00:53 PST

My credit card company is annoying me. I've gotten probably 10 calls in the last 3-4 months ostensibly from Chase offering me a new feature on my Credit Card. Basically they want me to purchase income insurance from them - if I pay them a few cents a month they'll allow me to carry a balance with no payments for a few months in the event of an unexpected job loss, medical emergency, etc.

All that is fine - the plan might be useful even. What's annoying is that the calls are frequent, repetitive, and deceptive.

The sales pitch basically goes like this: As a valued customer we're offering you a feature on your plan and we'd just like to send you an informational packet about it. Now if you'll just confirm your home mailing address for me...

Wait a minute - I get a statement from Chase every month, why would they need to confirm my address?

As it turns out I'm dealing with a salesperson who needs "consent" to add the feature to my plan on a trial basis and if I give my address that's taken as consent to turn on the feature as well!

I actually got one caller to admit that if I gave her my address she was going to make a change to my account.

I consider this to be deceptive. I recognize that the companies I do business with can call me or mail me sales information. I don't like, however, tactics designed to claim consent in ordering services without actually telling you what's going on. Consider this a warning when choosing credit cards...

Baypiggies July Meeting, etc

Sat, 14 Jul 2007 12:00:35 PST

I went to the monthly Bay Area Python Interest Group meeting again on Thursday night. The scheduled Python Newbies talk by Alex Martelli got pushed back and instead we had 3 presenters discussing the Natural Language ToolKit. This turned out to be very interesting - NLTK is an attempt to build a standard library of data, datastructures, and algorithms for natural language processing code. It's all written in Python and includes things like tokenizers, parsers, stemmers, and easy ways to train algorithms on well known data sets. Some of the presenters demoed code and some explained the general field of Natural Language Processing. It all sounded suspiciously close to AI to me - tasks like translation between languages was what I was thinking about going into it but it sounds like the targets are more like summarization and statement equivalence checking - things that come very close to requiring understanding.

As an example one presenter showed a sample sentence of florid sportscaster prose about an Italian skier who "captured two gold medals at the Calgary winter games". An end goal of NLP is to then be able to summarize that sentence down to "Alberto Tomba won a race". This goes past grammatical parsing (object of the sentence, verb action, etc) and delves into context and factual equivalence (understanding that "captured two gold medals" means that he won, for instance). These are lofty goals!

I also had fun taking a guest to the session. Last weekend was my wife's extended family reunion and the California branch of the family was playing host. The metawife and I fixed baked beans and BBQ chicken for ~80 people last saturday (with a grill assist from my dad!) as part of a progressive dinner. Everything went well (and the strategy of BBQing chicken till 80% done (like 150%) and then putting in foil lined ice chest to steam finish results in delicious moist chicken).

Anyways - back to the guest - one of the extended relatives stayed in town a week on vacation and turned out to be a programmer with a CS degree from Wright State in Ohio. I didn't actually get a chance to meet him during the reunion but offered to bring him to Google and enjoyed talking shop along the way. Now, I regret to say, he's bummed because he's going to have to go home and add Python to the list of languages he needs to learn. Nothing like making a little language advocacy part of your family reunion!

Sundries

Wed, 04 Jul 2007 00:50:15 PST

Welcome back Cyrus! My brother returned from his year long stint in Chicago (for training) and then Brazil and Paraguay. He's a lot thinner, has a little more Spanish and Portuguese than he used to and drinks terere - sipping cold water poured over yerba mate through a special metal straw with a filter on the end... This is not the worst habit involving "grass" and pipe-like implements he could have picked...

The other brother was over here today - discussing his recent music and book reviews I realized I really have a thing for genre bending music - Flogging Molly for your basic hard rock with traditional Celtic instruments, Dirty Three when you're in the mood for a power trio (violin, guitar, percussion) that sounds like raga music with the amp turned to 11, and of course the inimitable Matisyahu has all your Hasidic-Reggae needs taken care of...

Who I'm really into at the moment though, is Eisley. Five (homeschooled, church worship band, etc) musical siblings and a bass player (wait - that didn't come out right... No, no actually that's fine) with ethereal voices, nice indie-rock instruments and sci-fi/literarily influenced lyrics. Check it out: