Dr. Jerry Pournelle

Email Me


Why not subscribe now?

Chaos Manor Subscribe Now



Useful Link(s)...

JerryPournelle.com


Hosting by
  Bluehost


Powered by Apache

Computing At Chaos Manor:
October 6, 2008

The User's Column, October, 2008
Column 339, Part 1
Jerry Pournelle jerryp@jerrypournelle.com
www.jerrypournelle.com
Copyright 2008 Jerry E. Pournelle, Ph.D.


As I predicted in the last column, there has been a massive financial reconstitution bill — often mistakenly called a "bailout bill" — intended to restore confidence in the financial system. It began as a four page proposal to inject liquidity into the system by allowing the Treasury to buy "toxic" mortgage based security derivative packages at something like their intrinsic price, through a mechanism of "reverse auction". A reverse auction works this way: the government offers to buy at a given discount price; the sellers accept or do not until there is no more action. The government then decides whether to raise the price and if so by how much, and another round begins. This continues until the worst of the bad paper is removed from the system.

This was intended to have the minimum government interference in the market while taking the bad debt paper out of the system; and if all went well, even though an enormous amount of money would be involved, the actual long term cost might be very low — and if things went well enough, the cost might be negative, that is, the government might make a profit on the deal as real estate prices stabilized and market forces prevailed.

If during the recovery there were further crises, the government could take over failing banks. That is more or less what Sweden did in a time of financial crisis some years ago; (link) the Swedish action was considered quite successful, and in fact Sweden made money on the deal while squeezing out most of the equity holdings of stockholders and executives who caused the crisis in the first place.

That was the original four page plan. Something similar but expanded to 100 pages was brought to Congress and defeated in the House of Representatives, which then adjourned for High Holidays. Then the Senate took up the work, and approved a 451 page plan that was adopted the next day by the House. Precisely what those 451 pages contain isn't clear. There may be a few Members of Congress and Senators who read all 451 pages — I guess — but not many, and it's unlikely that they also read all the documents referenced and in some cases were incorporated by reference. We do know that this bill, said by some Members of Congress to be the most important legislation they had ever voted on, contained provisions to repeal taxes on wooden arrows, a subsidy for those who commute by bicycle, and various "green" provisions. God knows what else is in there. I expect it will be years and some court decisions before we know the last of what this bill does.

Of course the intended purpose is to unclog the financial system by soaking up a great deal of paper whose worth cannot be established because there isn't any market due to uncertainty. The uncertainty is due to the complexity of the packages. I'll make up an example of a typical package: You buy a paper that delivers to you 4.5% of the income from a mortgage on a house that sold for $500,000 in 2004 to a person with a stated but not verified income of $75,000 a year (no down payment) who has not missed any payments; 9.6% of the income from a house that sold in 2000 for $300,000 to a person with a verified income of $67,000 a year and a down payment of $45,000, who has missed one payment since made up; 11.23% of the income from a house that sold in 2007 for $450,000 to a person of no stated income no verified assets who has made no payments whatever and is now 3 months in arrears; 28.6% of the income from a house that sold in 1970 and was refinanced three times over the years to a family that has never missed a payment — well, you get the idea. There would be perhaps a hundred of those in the package. Each one of the hundred is different in price paid, purchaser qualification which can range from verified income with down payment to "stated income and unverified assets". The total of your share of the moneys paid by purchasers (and thus the nominal worth of the assets backing the package) is, say, $30 million dollars. You paid $29 million for the package. Statistical tools predict a maximum default rate of 4%, based on an average defaults of loans over the past 30 years. About 6% of the mortgages are in default now, and more are threatened. What will you sell this package for?

The point here is that no one truly understands the situation: it may be worse than anyone knows, it may be a lot better than anyone suspects. What's important to understand is that we do have skills, we have factories, we have a computer in every home and in every office and in every classroom. The fundamental wealth of the nation hasn't changed much — and the government has shown that the American pragmatic streak has not ended. If this move doesn't stop a new Depression, we'll try something else until something does. That, at least, ought to inspire some confidence — and confidence is what's lacking at the moment.

The High Tech Disaster

Note that this whole mess would be impossible in the days before there was a computer in every office. No one could have devised those derivatives without computers.

Last week I concluded that whatever happens next, we can be confident that there will be a sharp decrease in research and development funds, and the result of all this is almost certain to cause a recession; readers ought to be thinking hard about cutting expenses and looking to their assets (both in capabilities and property). I have no reason to change either conclusion, but then it shouldn't take a crisis to persuade people to look into their potentials.

The lack of research funds isn't going to slow the development of hardware; that's been done, and Moore's Law will march on for a while no matter what. Since our hardware is so far ahead of our software, there are still plenty of opportunities for computer savvy people. Machines are faster, and no one has to learn incomprehensible computer languages. Good, strongly structured languages, with strong typing and range checking, are widely available, many of them free. The main feature of such languages is that the compiler catches the errors. This makes debugging a great deal easier. My point here is that if you know of something a computer should be doing, it's a lot easier to learn how to teach the machines to do it than it was back in the early days when it took wizardry to write efficient programs and debug them. Today's computers have so much memory, and mass storage, and are so fast that efficiency isn't very important.

One more point: historically those who bought and held after the crash of 1929 did extremely well; but by held I mean held for years. They had money for investment that they did not need to live on, and could afford to hold their investments for a decade and more. On the other hand, there needs to be some intelligence to the analysis one makes. One chap studied the fundamentals, decided that the future was in automobiles, and invested everything he had in Hupmobile.

Languages to Begin With

If you want to try learning what programming is like, I recommend you start with Python. (Python.org) It's free, it's fast, and there are a lot of example programs you can look through. The Wikipedia article on Python gives a good description of the language and its history, as well as an example of a Python program. Don't let the capability of the language fool you into thinking you need to learn a lot before you can do anything: you can write simple good programs in Python within a couple of hours of beginning. O'Reilly has several books on the language; I recommend Learning Python as a beginning, and those more serious about learning the language need the O'Reilly Python Cookbook — at least if you are like me and more comfortable learning languages by studying examples.

Peter Glaskowsky notes that "Python is a language that relies on dynamic typing and other kinds of looseness in order to increase programmer productivity," and may not be entirely suitable for learning good programming practices. I have to agree, but it is free, it does work, and I confess I use it when I have a job that needs doing fast. In my case I often craft filters and other specialized text processing, and Python is excellent for that. I've never attempted to write a large and complex Python program.

For those with no programming experience with strongly structured, strongly typed languages (or any programming experience at all), many — including me — recommend beginning with Pascal. There is an excellent Pascal tutorial that will also point you to a free Pascal compiler. Pascal was originally written by Niklaus Wirth of ETH (Eidgenössische Technische Hochschule often known as "the Swiss MIT") as a programming language; this was well before the days of small computers, and Pascal was written as an instructional language; indeed, when Wirth wrote Pascal computers were sufficiently rare that most instructors wouldn't have one, and would have to judge the worth of student programs by reading them and "head compiling". This requirement had the great side effect of making Pascal a great deal simpler and more comprehensible than other computer languages.

In the early days of small computers there were a number of companies, including Microsoft, that developed Pascal compilers. The best known and most useful of these was probably Borland's, and indeed the ETH Medical School was using medical diagnostic programs of 50,000 lines (later more) of Borland Turbo Pascal when I visited Zurich to interview Dr. Wirth. Turbo Pascal evolved into Delphi of which Delphi 2009 is the latest version. I have no experience with the latest versions of Delphi, but the Windows version of Mrs. Pournelle's reading program was written in an earlier version of Delphi and has continued to work through several versions of Windows.

For those who want to learn programming, study of Python and working through the Pascal tutorial will get you started without much expense, and the principles you learn will be useful for learning nearly any language. I do warn you: neither Python nor Pascal is considered an "efficient" language, and many who write in "real systems development languages" such as C and C++ are openly contemptuous of Pascal programmers. Years ago my friend and sometimes colleague Marvin Minsky once said of my preference for strongly typed languages "He thinks it's better to put on a straightjacket before he goes to work." Marvin can program in nearly any language known to man (and a few I would swear were known only to aliens); at that time his preference was for LISP although I also saw him working with APL. In those days language efficiency was very important because computers had severe limits in memory, disk storage, and speed.

Things are different today; but even then there wasn't as much cost to using highly structured (and thus "restrictive") languages as many thought. It might take longer to write a program in a highly structured language and it certainly takes longer to compile it, but the time from start of programming to a working application is what one ought to be concerned with: and debugging unstructured languages can take a lot longer than anyone thought. They are also more vulnerable to hacks and attacks: for example, programs that do range checking at compile time aren't generally subject to stack overflow hacks.

This will probably open a new round of the language wars, but do note I am mostly addressing those who are thinking of learning to program.

The Reliability of Wikipedia

Wikipedia is enormously useful; for its history and purpose see the Wikipedia article on Wikipedia. In theory all Wikipedia entries can be edited by anyone; vandals can trash an entry but fairly quickly someone will put it right again (if only by restoring what used to be there). Partisans who unfairly change an entry are similarly policed by partisans who disagree; theoretically the entry will evolve toward something sufficiently objective that everyone will leave it alone.

Other entries aren't so controversial. Science and technology articles are often read by those most expert in the field. At least one study has reported finding fewer errors in Wikipedia than in the on-line Encyclopedia Britannica.

For better or worse, Wikipedia is with us, is often the first place one looks for non-controversial information, and some use it as their only real source on the theory that it's as good as anything else. On the other hand, sometimes there's a more sinister side to the discussion.

I refer you first to the Forbes article on short selling and then to the Register story on how manipulating Wikipedia helped bring about the market disaster. I doubt one can prove the assertion, but it's certainly intriguing. I also doubt we have heard the last of this story.

Disk Drives and Storage

Roxanne was built in late 2005 and later upgraded in 2006 to run Vista Release Candidate 1. At the time she was built, the latest and greatest disk drives were Seagate 160 GB drives. I put in two, one for backup. The primary 160 GB didn't quite fill up, although it got close; but those drives served me well, and continue to do so. Roxanne has been my main writing and gaming machine for a couple of years now (it was just replaced by Emily), and the master copies of all works in progress resided there. Moreover, Roxanne went through all the Release Candidates of Vista, several versions, and all the updates. The system wasn't very stable at first, but that wasn't the hardware's fault. As I said, those Seagate drives have been working hard for years, and I've never had a glitch.

When it came time to build Emily, the Intel Core 2 Quad Extreme system (described last month) we used Western Digital Caviar 1 Terabyte 16 MB cache 3 GB/s drives. (link) I am still astonished at this: a full terabyte! I was tempted to set up a RAID system with those drives, but I didn't do that: none of my operations are slowed by a "mere" 3 gb/s (which translates to 300-500 mb/s throughput). The Caviars are plenty good enough for what I'm doing.

My next move will be to use Norton Ghost to transfer everything, operating system and all, from Roxanne's 160 GB main drive to a WD Caviar 1 terabyte drive. I've done that before, so I anticipate no difficulties at all. Of course an even simpler move would be to put a terabyte drive in as the secondary and transfer all data files to that. Emily has two 1 terabyte drives; I wonder what her replacement will have?

It's not quite a disk drive, but the other day Woot! was selling Kingston 2 GB Micro SD (with full size SD carriers if you need that form factor) for $5. I bought several of them to use with my Lumix digital camera. Now if I can find a similar deal on Memory Sticks for the Sony DSC 1100... Everywhere you turn, and in every form factor you can imagine, storage is getting cheaper. Hurrah.

While I am at it, I also have coming a Scorpio WD 3200 BEVT laptop drive to install in the Mac Book Pro. I will probably have that done by the Apple Store to avoid voiding the warranty. More on that as it develops. WD is coming out with a 500 GB Scorpio Blue drive, and Samsung sells a 500 GB laptop drive for $179 on Amazon, so there's no lack of disk capacity for laptops. Since I can't replace the Mac Book Pro drive myself, I am mildly tempted to get a 500 GB drive on the theory that I'll want to run Vista in addition to Mac OS-X on the Mac Book Pro; but we'll see. The 330 MB drive is really likely to be enough given that I only have 120 MB on the ThinkPad t42p and I've never felt squeezed for space. I use laptops as laptops, not as desktop replacements. That could change, of course.

Mac and Vista Networking Tricks

My network operates under Windows 2000 Server. I have a system ready to accept Server 2008, but for various reasons I haven't got around to switching over. So far Windows 2000 Server Active Directory has worked very well for me. It allows me to let Roberta have a "power user" account that doesn't interfere with her operations on her own machines, and gives her access to some of the other machines in the system, without giving her the capability of doing any real damage to the system. At the same time I have much broader permissions under my own user name. In my case I log on as username in the domainname domain, and all is well.

The Mac has no trouble seeing Windows systems in the network. Logging on to them is a bit more difficult. Naturally I log on to the Mac with my Mac username and password, and those are different from the username and password I use for the main network. When I use the Mac to log on to a system running Windows XP, it asks for my name and password. I give it the same user name I use on the Windows network, give the password, and all is well.

When I try to connect to a Vista system from a Mac, I can see the system and all its drives. When I try to log on to a Vista system and give username and password, I am told I can't do that. I don't have permission. This very nearly drove me nuts until I figured out what I have to do.

To Qualify or Not to Qualify

Most people neither know nor care whether their network domains are fully qualified, but in this case it matters. When I first set up my system I gave the network a networkname; but I was persuaded that I should set up a "fully qualified domain" by making the formal name networkname.jerrypournelle.com. So long as I was running Windows 2000 or Windows XP this made no difference whatever. It didn't even matter when I added Vista systems to the network; but when I added the Macs (Mac Book Air and iMac 20) to the system, qualification became important.

When I go to the Mac and want to log on to a shared drive (or folder) on a Vista system, I have to (at the Mac system) use username@domainname.jerrypournelle.com as the username when I access a Vista machine from the Mac. The password remains the same. And Lo! All is well!

This actually makes sense, but it took me a while to figure it out. Experienced Mac users tell me they didn't know it either.

New Kindles and Sony eBook Readers

I haven't seen either one of them, and the rumor is that the Sony with touchscreen will cost $400 in the US so I am not really tempted to replace my Kindle 1 with that; but the new eBook readers are coming. (Sony: link) (Kindle 2 link)

What's interesting here is that eBook Reader sales are apparently good enough to keep both Sony and Amazon working on improvements; and my experience with the Kindle is that it's already Good Enough for much of the reading I do. At least half of the books I buy now are for the Kindle.

Eventually we're going to have color eBook readers with satisfactory battery life sold for under $50; before that happens, the nature of the publishing business will change. When eBook sales are large then the reader prices will fall even more sharply. We're looking at the beginning (actually we are well past the beginning) of a new revolution here, and before it's over it's going to have effects comparable to the computer revolution. (Long time readers will recall that my column in McGraw Hill's Byte companion Popular Computing Magazine was titled "The Computer Revolution").

In the early days of computers, the advances came so fast that it was difficult to decide when to upgrade; it looks as if we're about to get to that stage with phones, tablet PC's, and book readers. Judging when to upgrade is going to be difficult for the next few years.

Eventually though, and sooner rather than later, well before this revolution is over, everyone will have a pocket computer much like the one described in Mote in God's Eye which Niven and I wrote in 1972. It will have a video camcorder, audio recorder, still camera, full voice and data communications, full Internet, full computer capability, and hundreds of books which can be read or listened to at preference, all in a form factor that makes it both easy to carry and to use. Exactly when that happens I am not sure, but I expect to live to see it.

Reviews

Statistics in a Nutshell, A Desktop Quick Reference, by Sarah Boslaugh and Paul Andrew Watters, O'Reilly. The subtitle of "a desktop quick reference" should be taken very seriously; this is a handbook, not an instruction manual. There are chapters that introduce and explain concepts such as relational data bases, but I really would not recommend this as one's introductory text on statistics, statistical inference, and the application of computers to statistical manipulation.

On the other hand, it's an excellent reference book, and for those who already know something about statistical models and how they work this is a highly useful work. I sure wish I'd had this as a refresher handbook back when I was in graduate school. Just don't use it to learn everything you're going to know about using statistics to make predictions.

Statistical inference works by assuming that certain mathematical functions describe reality. We can then manipulate the models to make inferences; for example, we can use measurements taken on randomly selected samples of the population to estimate with varying degrees of accuracy the real values we would find if we tested the entire population. We can, for example, take the average of the values we obtain in the sample and with great accuracy estimate the true mean of the population and the probability that the actual true mean will differ from our estimate by any given amount. We also use statistics to determine the outcome of experiments: if we grow crops using different treatments, what is the probability that the differences in yields we get are due to the treatments as opposed to those differences arising by chance? And of course psychologists use statistics in hopes of interpreting most of their experiments.

The O'Reilly Statistics in a Nutshell will explain how statisticians do this sort of thing, and if that's what you need to do, this book is as good as any I have seen and a lot better than most. It explains techniques and methods, formulas, and other such matters that anyone working with statistics needs to know.

What it doesn't explain is that the map is not the territory, and our assumptions of the distribution of data in the real world are terribly important. As a trivial example: suppose we want to determine the true mean (average) American wealth. We go choose subjects at random and determine their wealth. After we have a sufficient number, we can use statistical techniques to determine that, given the number in our sample, the true mean wealth differs very little from the average of our sample. Moreover, every new (randomly chosen) subject we add to our sample gives us greater confidence in our estimate.

Now we add one more subject to the sample, but this one wasn't chosen at random. We add Bill Gates. At this point our average becomes meaningless. So do most of our other measures. (Not all: the median doesn't change much by adding Gates; but everything else does.)

That example is trivial, but the concept behind it is not. As Taleb notes in his book The Black Swan (reviewed here) most of our statistical techniques do not properly take account of improbable events. Our statistical models — maps — are not the territory, and if we falsely assume that they are we often experience a terrible awakening.

It is of course improper to condemn a book for not being something it never pretended to be. Statistics in a Nutshell is excellent on presenting analytical methods, and gives details on techniques not well covered in other books. Like all O'Reilly books it is well edited, has few typographical errors (I actually found none) and has a well designed layout. As I said above, I can recommend this book to anyone who has to work with statistics.

For those who do have considerable experience using statistical analysis, I recommend two more books: Taleb's Black Swan, and Leonard J. Savage, The Foundations of Statistics (Wiley, 1954). Savage's book is tough reading. It employs a number of equations, and on first reading it's often difficult to understand what Savage means to say; this is one of those books that have to be read twice, and possibly more often, as Savage goes into what we mean by probabilities and uncertainties. In a game like craps there are no real uncertainties. Yet, actually there are, but they are external to the game, as for instance when the casino brings in a breakout artist to introduce loaded dice to the game. In the real world there are real uncertainties, and we have no statistical way to estimate the degree of greed and the willingness of casino operators to take risks. We can only assume they are very low in a licensed casino, might be a higher in a "friendly" private game, and higher still in a floating back alley crapshoot.

Fair warning: I expect that considerably fewer than half my readers will ever get through Savage, so unless you are very interested in this subject you shouldn't bother. That's not true of Taleb's Black Swan and Statistics in a Nutshell.

Those who must create presentations but don't do it very often will get a lot out of Nancy Duarte's slide:ology The Art and Science of Creating Great Presentations. I fall into that category, and I certainly did. Duarte is Principal of Duarte Design, a well known Silicon Valley consulting firm. The book is more about concepts and expanding your view of how presentations work than about techniques. On the other hand, you don't need a book on techniques. Most design problems are more conceptual than technical. Recommended.

I have been enjoying Janet Evanovich's series about Stephanie Plum, a rather inept bounty hunter who lives in "the burg" in Trenton, New Jersey. The series begins with a book entitled One for the Money. Like many modern popular crime novels this is told in first person by a female character, and the entertainment is more character than plot. Stephanie Plum is certainly a character: she's attractive, lives alone with a hamster for a pet, and is perpetually in danger of going to her parents' place for dinner only to find that her mother has arranged a companion to join her. Meanwhile, Stephanie is trying to make a living, and not doing very well at it; in One for the Money, she gets involved in apprehending bail jumpers for her Cousin Vinnie who runs a bail bond company. Fortunately, Stephanie has friends — strong and competent male friends — who are willing to help her, and she's not too proud or feminist to ask for help. (Indeed she rather imposes, to the point where Mother Plum thinks one of the helpers is a boyfriend.) The rest of the series shows Stephanie sort of learning her trade while avoiding insanity. Good light reading. Recommended.