See Greg learn.

Gaussian FAQ

2010-04-01T07:23:00.000-07:00

I found this on my computer and can't seem to find it on the web, so I'm putting it here in case other's would benefit. Note: I didn't write this.

====================John D'Errico, 24 Mar 1995=======ssm
From: John D'Errico 
Subject: Gaussian FAQ
Message-ID: <3kv7m3$61f@thetimes.pixel.kodak.com>

This is a FAQ which attempts to cover questions about the normal
(Gaussian) distribution. I will attempt to add or enhance topics
as I have the time and people make suggestions.

Questions to be answered in this FAQ are:

1. What is the normal (Gaussian) density function?

2. What is the cumulative normal distribution function?

3. How is ERF related to the cumulative normal distribution function?

4. What is a good approximation to the cumulative normal function?

5. What is a good approximation to the inverse of the cumulative
distribution normal function?

6. How do you generate a normally distributed random variable?

7. If X is a normally distributed random variable with mean = 0
standard deviation = 1, how do you convert it into one with mean = mu
and standard deviation = sigma?

8. How do you generate bivariate or multivariate random normal
variates with a specified correlation matrix? Covariance matrix?

9. If X is a normally distributed random variable, what is the
distribution of f(X)?

10. What are some good references on these topics?

======================================================================

Answers to questions 1-10:

1. What is the normal (Gaussian) density function?

The standard (mean=0 and variance=1) normal density function is

             1       -(1/2)*x^2
 Z(x) = ----------  e
        sqrt(2*pi)

For mean=mu and variance=sigma^2
 [ correction, inserted (1/2) below; 17-aug-00/rfu ] 

                1          -(1/2)((x-mu)/sigma)^2
 Z(x) = ----------------  e
        sqrt(2*pi*sigma)

The normal distribution with mean=mu and variance=sigma^2 is
customarily referred to as N(mu,sigma^2). The standard normal
is N(0,1).

----------------------------------------------------------------------

2. What is the cumulative normal distribution function?

The cumulative normal, referred to here by P(X), is simply the
integral of Z(x) dx from minus infinity to X.

We will use Q(X) to refer to the integral of Z(x) dx from X to
plus infinity.

P(X) + Q(X) = 1

----------------------------------------------------------------------

3. How is ERF related to the cumulative normal distribution function?

Since some systems do not provide a cumulative normal function but do
provide ERF, this can be useful if you know the transformation between
them.

ERF(X) = integral [2/sqrt(pi)]*exp(-x^2) dx from 0 to X.

P(X) = [ERF(X/sqrt(2)) + 1]/2

----------------------------------------------------------------------

4. What is a good approximation to the cumulative normal function?

While it is not possible to write a solution to this in a simple,
closed form expression not involving infinite sums, there are many
good approximations available. It is not necessary to resort to
numerical integration techniques such as trapezoidal or Simpson's
rule. In fact, far more accuracy can be achieved through one of the
approximations which can be found in the references listed below.

One simple approximation to P(X), the cumulative normal, is:
(from Abramowitz and Stegun)

Let t = 1/(1+pX)

where p=.33267

then

P(x) = 1-Z(x)*[a_1*t + a_2*t^2 + a_3*t^3]

where

a_1 =  0.4361836
a_2 = -0.1201676
a_3 =  0.9372980

The error in this approximation is less than 0.00001 for all X.

Other approximations can be found in Abramowitz & Stegun; Press, et al;
and in Hart, et al. In particular, Hart has a very extensive set of
approximations, some with accuracy over 20 significant digits within
a given region.

----------------------------------------------------------------------

5. What is a good approximation to the inverse of the cumulative
normal distribution function?

Mathematically we can write this as:  Find X such that Q(X) = p for
any 0 < p < 1. (Note: this computes an upper tail probability.)

Again, it is not possible to write this as a closed form expression,
so we resort to approximations. Because of the symmetry of the normal
distribution, we need only consider 0 < p < 0.5. If you have p > 0.5,
then apply the algorithm below to q = 1-p, and then negate the value
for X obtained. (This approximation is also from Abramowitz and
Stegun.)

t = sqrt[ ln(1/p^2) ]

             c_0 + c_1*t + c_2*t^2
X = t -  ------------------------------
         1 + d_1*t + d_2*t^2 + d_3*t^3

c_0 = 2.515517
c_1 = 0.802853
c_2 = 0.010328

d_1 = 1.432788
d_2 = 0.189269
d_3 = 0.001308

See Abramowitz and Stegun; Press, et al.

----------------------------------------------------------------------

6. How do you generate a normally distributed random variable?

There are several good ways to do this, some better than others.
It is assumed that the user has available a good uniform pseudo-
random deviate generator available. This is the starting point
for all such approaches.

The first algorithm is to make use of the central limit theorem.
This says that the sum of a set of independent, identically
distributed (commonly abbreviated as iid) random numbers will
approach normality as enough of them are added together.

So, simply generate n uniform [0,1] random numbers, and sum them
up. Then shift and scale this sum to have the appropriate mean
and variance to give us a roughly N(0,1) random variable. Thus,

    sum(X_i) - n/2
Z = --------------
      sqrt(n/12)

Note that if we choose n = 12, we need not even do a divide,

Z = sum(X_i) - 6

While the method described above is quite simple to program, it
does a poor job of generating random deviates in the tails of the
distribution. There are far better ways to do accomplish this
task. A logical way to generate a normal random deviate is to
look back to the answer to question (5).

Simply generate a single uniform [0,1] random deviate X, and then
invert the cumulative normal at that point. If the inverse is an
accurate one, then that number will be a pseudo-random normal deviate.
This method will be as fast and as good as is the approximation
you choose for the inverse of the cumulative normal.

The other common method, which generates pairs of normal deviates
at a time (or if you want, bivariate normal deviates) is the Box-
Muller transformation. See Numerical Recipes (Press, et. al.) for
an explanation of this method.

----------------------------------------------------------------------

7. If X is a N(0,1) distributed random variable, how do you convert
it into N(mu,sigma^2)?

Referring back to (1), we see that a simple transformation will
accomplish this.

Z = mu + sigma*X

----------------------------------------------------------------------

8. How do you generate multivariate random normal variates with
a specified correlation matrix and set of variances?

Lets start with a simple case, just a bivariate normal. We will
assume s1 and s2 are the standard deviations, c is the desired
correlation between them (-1 <= c <= 1) and mu1 and mu2 are the
desired means.

First generate two unit normal random deviates, N1 and N2.

  X = mu1 + s1*N1

  Y = mu2 + c*s2*N1 + sqrt(1-c^2)*s2*N2

X and Y will be bivariate random normal deviates with the desired
distribution parameters.

Now for the more general case of a multivariate normal where "multi"
means larger than two.

Suppose we have a correlation matrix C. We need to convert this to a
covariance matrix, S, first.

Let D be a diagonal matrix with the desired variable standard
deviations on its diagonal. Then

S = D C D

Form the Cholesky factorization of S (actually any matrix "square
root" of s will do.) This is a factorization of S in terms of a lower
triangular matrix L such that

S = L' L

This factorization always exists whenever S is positive definite
(non-singular in this case, since we started from a supposedly
valid correlation matrix.)

Generate an n by p array, A, of N(0,1) random deviates, where p is
the number of variables and n is the number of multivariate sample
you wish to end up with. Then

B = A L'

is an array of normally distributed random deviates where each row
is an event with roughly the correlation/covariance matrix you wanted.
(It won't be exactly that because of pseudo-random variability.)

Once you have generated a set of random deviates with the desired
correlation matrix and mean vector zero, simply add a vector offset
to obtain any desired mean.

If the correlation/covariance matrix supplied is singular, a similar
approach can still be used, but now we must define L using a different
type of matrix square root. This can be achieved using an eigenvalue
decomposition of the matrix S.

----------------------------------------------------------------------

9. If X is a normally distributed random variable, what is the
distribution of f(X)?

There are several simple cases we can cover. If X and Y are N(0,1)
independent R.V.s, then

aX + bY is a normal random variable for constants a and b.

X/Y is a Cauchy random variable (and its moments are undefined.)

e^X is a lognormal random variable.

X^2 + Y^2 is a Chi-squared random variable with two degrees of
freedom.

sqrt(x^2 + y^2) is a Rayleigh random variable.

These distributions along with many others can all be found in
Johnson, et al.

The general problem of determining the distribution of a function of
one or more random variables is known as statistical tolerancing.
This can be done by several techniques, Monte Carlo methods, first
or second order Taylor series approximations, and Taguchi style
methods (actually related to Gaussian quadrature) to name a few.
A common approach it to estimate the first few (generally either
2 or 4) moments of the transformation in question and then find a
distribution which has those same moments. A family of distributions
which can be used to match the first 4 moments is the Pearson system
(see Johnson, et al.)

See D'Errico & Zaino for a discussion of statistical tolerancing.

----------------------------------------------------------------------

10. What are some good references on these topics?

The references listed here are not necessarily the best possible ones
for this subject.

  1. Abramowitz and Stegun, "Handbook of Mathematical Functions",
     Dover Publications, 1965 (originally published by the National
     Bureau of Standards, 1964)

  2. Press, Flannery, Teukolsky, Vetterling, "Numerical Recipes",
     Cambridge University Press, 1986

  3. Hart, et al, "Computer Approximations", Wiley, 1968

  4. Johnson, Kotz, & Balakrishnan, "Continuous Univariate
      Distributions", Volume 1, Wiley, 1994

  5. Kalos & Whitlock, "Monte Carlo Methods", Wiley, 1986

  6. Knuth, "The Art of Computer Programming, Volume 2, Seminumerical
      Algorithms", Addison-Wesley, 1969

  7. D'Errico & Zaino, "Statistical Tolerancing Using a Modification
      of Taguchi's Method", Technometrics, Vol 30, no. 4

======================================================================

John D'Errico,  derrico@pixel.kodak.com

Any errors or omissions are my fault and mine alone.

======================================================================

Where's my $HOME?

2009-01-31T14:01:00.000-08:00

Once upon a time, life was simple. My progress could be measure by the state of my home directory. My work lived in folders that represented input or output data, program source code, and latex source code. Beyond that there were some odds and ends like gnuplot script files and the resulting postscript files, either to analyze or include in whatever paper I was writing. I also had a Mail directory that my email client (elm at first and then pine) access, but I didn't really use email that much. I would use emacs (or vi if it was a quick edit) to edit a file or run my programs to generate output data. That was pretty much it. I lived in the shell. I could telnet from wherever and have the same environment from wherever I was. As time went on, I found myself using emacs more and more as my shell, but the idea was basically the same.

Now... I usually have email, a web browser, and some office app or adobe open, usually to read, and sometimes to write. I'll be editing code sometimes (less often then I like) and have used pythonwin, but recently I've rediscovered emacs (now on windows). Instead of constantly having an awareness of my directory structure, because I'm cd'ing around it in the shell, it's mostly a place to store some program's files or downloads from my web browser. It ends up getting messy because I'm not as cognizant of where everything belongs. The file system is just a dumping ground.

To make matters worse, I have stuff on multiple computers (home, work, and laptop) and I have stuff that just lives on the web (delicious, twine, google docs, evernote, whatever I'm intrigued by this week). I'm homeless. Where's $HOME?

Holy cow. Where am I? How's my progress? What's my operating system doing for me? I used to live in the shell. Todo management? "grep -R 'TODO' ." I'd make up scripts to handle common tasks and throw them in ~/bin. A lot of that was made possible by the miracle of plain text. Unfortunately, I deal in binary files (.doc, .ppt, .pdf, .xls, etc.) these days and I have to open some separate window to look at them.

50th time is the charm with GTD?

2009-01-03T17:02:00.000-08:00

Well, it's a new year and I, like many, and making an effort to organize. I've read David Allen's Getting Things Done and been inspired, but have never quite got a system I'm terrifically happy with. Usually, I try something new, move all my lists, get a temporary boost (more from inspiration than efficiency) and then things end up about the same.

Well, this is another one of those times! This post from March sounds startling like this one, but I will press ahead. The idea of using google notebook didn't last long. I'm not sure why. I think part of it is the interface. It just has all these borders and stuff that get in the way of the lists. I fell in love again with emacs and after using org-mode for outlining, put my lists in there and used svn for versioning. I found myself reordering branches, indenting, and generally messing with my giant .org files (work and home). Worse, I found I was generally doing things that never made it on the list and not looking at the list to see what to do.

Next (yes, there's a next), I tried taskodone, which based on taskpaper. I was inspired by this review of taskpaper

"TaskPaper’s strength is that it lets you focus on crossing out those tasks instead of building a self-referential web of unfinished business which separates you from the cold, harsh reality of all the work you need to do." — Scott McNulty, Tuaw.com

Hey, that's me! OK, so I tried taskodone and I like it, and I really think it might be OK. It has a lot of charm in that it's pretty fiddle proof with only one level of indendation. It didn't quite stick with me though. I still wanted to reorganize my unfinished business. I also didn't like looking at it all all the time. I suppose some discipline, like just looking at the @na lines would work, but still when I opened it to delete a finished line or add a next task, I'm still looking at everything.

All of the above with tags and outlines actually has little to do with what's in David Allen's book. His lists are just lists. Many of us have this need to keep actions tied (electronically, preferably) to their projects. His project list though, is just a list of projects! You only consult it at the weekly review or when adding a new one. The other lists or lists contain your next actions.

Where does the planning happen? In the project plans, of course. These project plans are culled for actions in the weekly review. They go in folders in the book, I believe. You get out these manilla things, look and alter them, and then update your plain old flat GTD lists. I had never quite figured this out (and am still not sure I have).

Where are the project plans in my org-mode giant list? Well, they are *all* right in there. Yes you can collapse and expand and only look at what you need with some fancy emacs keystrokes, but still, they are *all* right there. Too much for me.

Where are the project plans with taskpaper? Well, umm, umm... There's a flat list of tasks under the project headings, but that's just not enough for a project of any complexity. You can break a project into separate project (which I did) and have something like "Build a house:", "Build a house - hire an architect," " Build a house - hire a contractor," etc. That's just cheating though, and your forced to look at all the sub-projects.

OK, so my projects don't really live in manilla folders too often. What could I use? Well, gee, how about those things on the filesystem we call "folders?" Wait, it's all coming back to me now. Back in grad school, I had a small set of folders that my life depended on. One folder would hold the .tex, .ps, and gnuplot script files that represent the paper (or my thesis) that I was working on. The other folder(s) would hold code and the output of that code. The state of those folder represented my progress in pretty absolute terms. When the paper was done, refereed, and published, I was pretty much done. I would put "TODO" or "FIXME" into code or tex files and have TODO files with lists of task. grep was my friend for finding where work was needed.

Much of that simplicity came from what I was doing and it was just natural. Now I'm more connected with other people and there's a lot of quick turnaround exchanges via email that can be confused with advancing the ball. Often there's not a product I'm actively working on that would naturally go in a folder. I might have one due in the distant future, but haven't actually got to the point of creating a folder and opening a word processor.

So here's the idea -- use project folders. Have a project? Make a folder. Put some stuff in there that represents what's been done and what has to be done. The state of these folders represent progress (or lack thereof). I'm thinking org-mode can help here with a TODO.org with the tasks. Keep TODOs separate from notes, and references. Flag the next action with TODO and use org agenda to pull the master action list together. The project list is just "ls ~/projects."

Really the only revelation (to me) is that having all your project plans in one file can be terribly distracting. OK. I've wasted enough time not actually doing things while writing this, so back to work!

proud papa

2008-10-30T13:11:00.000-07:00

I installed python on my 10-year-old's computer a while ago and showed him a few things. He's really into football now and for some reason he wanted to calculate NFL passer ratings for some QBs. I helped him write a program in python that prompts the user for completions, attempts, etc. and then prints out the passer rating. This morning I found him on nfl.com looking up stats and punching them into his new calculator. I think it's the first time he's created a tool. It feels a bit like watching him discover fire. I'm a proud papa.

good old SQL

2008-09-25T15:38:00.000-07:00

It's been a little while since I've done this, but somewhere along the way while I was building a dictionary of dictionaries, that what I was doing was creating a little database and that there was this language extracting data from such things. Plus, the data was already in an MS Access database, which I could query using odbc.

I think I can get about 90% of what I want with a well-crafted query. I could do the rest in python, but I think I could simplify things with a user-defined aggregate function. I think I could do that with Access but I don't know how, so I'll probably move the data to sqlite and then use that to define my functions. Knowing that sqlite trick is also probably a more reusable skill (for me at least).

The database came with lots of nearly impenetrable VB code that creates temporary tables and aggregates in VB instead of in SQL (using GROUP BY). To make sure I understand what they're doing (and the data) before I start generating my own reports, I'd like to be able to reproduce their results. It'd be a neat trick to turn thousands of lines of VB into a tenth that in SQL and python. Ulimately, it would be more transparent and maintainable as well.

defaultdict and R

2008-09-24T15:23:00.000-07:00

In answering a question on the python-tutor list, someone used collections.defaultdict. I didn't know about this. I guess it looks a little cleaner to say a = defaultdict(list) rather than use a.setdefault(key,[]).

Well, that isn't much learning for the day, but it's something.

I'm also trying to learn and use R, but fell back to python for my task today when I couldn't quickly figure out how to do what I wanted to in R. R is frustrating right now because it seems so insanely powerful, but at the same time foreign to how my python-oriented brain thinks.

I have "Modern Applied Statistics with S-PLUS" and learned to parrot the examples in order to do some cool stuff, but when I have to deviate much from the examples, I find myself a bit lost. I figure I don't understand the basics of the language well enough and now have "Programming with Data." Hopefully this will get me up to speed, so I can be one of those guys on the R-help list who can answer a question with six different one-liners to accomplish the same thing.

emacs org-mode

2008-09-24T06:54:00.000-07:00

I tried org-mode a while ago, loved it, and then inexplicably stopped using it. I think I was trying to figure out how to integrate my entire workflow (calendar, task, notes, etc.) in emacs and it was too much at once. This time I was just using it for notes/outlining and it is quite amazing. The keys are very intuitive also. You use the Meta and Shift keys combined with the arrow keys to move sections up and down, indent, etc.

I also used the table mode to actually make a table I needed and it is, quite simply, incredible. It's amazingly intuitive to fill a table, move rows and columns, insert, and delete. The thing can even be used as a spreadsheet, but I haven't tried that yet.

my fingers remember emacs

2008-09-16T16:00:00.000-07:00

Long ago, I used to live in emacs. ctrl-x-`, etc. I've been playing with pydev and eclipse recently and got frustrated with trying to figure out how to add modules to the python path that are "egg-link"s. So, I fired up emacs to start editing my text and I can't believe how much my fingers remember. I did a ctrl-5 2 to open a new window without blinking an eye. Where did that come from?

pytables - more meandering

2008-08-05T06:21:00.000-07:00

I've been playing with pytables the last couple of days. I like it a lot for its great efficiency. I was pretty disturbed when my sqlite file was 60+MB. It forces me into a hierarchical format though and I'm not sure how I feel about that. This is a situation where it really might be OK. Laps and tracks really below to a particular run. On the other hand, I could probably easily solve the sqlite inefficiency by storing the sample data as (possibly compressed) blobs.

javascript libraries - there are a lot of them

2008-07-20T16:17:00.000-07:00

And I thought the multitude of python web frameworks was bad. I was playing with google app engine and wondered if it had some library for making your app all AJAXy and cool. It doesn't.

Django doesn't have some preferred library. Turbogears 1 did (MochiKit), but I'm sure that (like everything else) has changed in Turbogears 2.

OK. I'll see what's out there. Holy cow. There are (off the top of my head) -- Mochikit, prototype, script.aculo.us, jQuery, dojo, mootools, ...

The breadth-first search continues...

meandering

2008-07-18T16:53:00.000-07:00

I posted on my running blog with the same title, so I thought I'd do it here also since it seems applicable. I picked up my program again and decided to work on importing my data from SportTracks. I thought it would be simple since the format is XML. It turns out the track data and heart rate data are in elements that contain blobs of base64 text. I asked on the SportTracks forum and was just told the schema was "changing." OK. I figured out the current format, but what if it changes?

SportTracks has a plugin architecture, so I thought maybe I'd look into that. It's in .NET and I like python, so I got sidetracked looking at IronPython and looking for books on .NET. What am I doing? Meandering.

For some reason, I got to looking at enthought tools again, specifically envisage. This got me to look at Eclipse, which is a java framework Envisage is based on. More meandering.

Then of course, I got back to wondering if a web app wasn't the best way to go, which brought me to revisiting the pylons, turbogears, or django question. In the midst of that, I discovered google app engine, which includes django templates. What was I doing again? Oh right, meandering.

on to pylons...

2008-03-19T17:11:00.000-07:00

I caught up a little bit with respect to turbogears happenings by perusing planet turbogears. Version 2 is going to be built on pylons, SQLAlchemy, genshi, and ToscaWidgets. Does that mean this book I have is basically worthless? OK, now I'm looking at pylons. I notice freebase.com is using pylons. That's a pretty good endorsement.

Multitude of python web frameworks are making my head hurt

2008-03-14T20:01:00.000-07:00

Django, Turbogears, Pylons, ... Oy vey. I already have a nice hard copy of the turbogears book I can read in bed. I think I'll just read that.

Turbogears dying?

2008-03-08T07:42:00.000-08:00

Well, with my ever-shifting focus, I decided to learn a little something about web development. I had this though before a couple of years ago and played with Turbogears, which at the time was the hot new thing. It was python's answer to Ruby on Rails.

I picked up the book, which I bought quite a while ago, yesterday and started working though the examples. I was a little frustrated they didn't work out of the box and in my googling for answers, I got the distinct impression that enthusiasm for Turbogears isn't what it used to be. What happened?

I guess I'll check out Django now. It seems to have more activity these days.

My nascent GTD system

2008-03-07T04:02:00.000-08:00

Getting Things Done is everywhere on the net. I read the book a few years ago and it's really very inspiring. Who wouldn't want "mind like water," but how making it actually happen is going to take some effort.

I've emptied my brain and put it all in lists many times and tried a ridiculous number of software programs, but haven't ever really got the hang of it, shifting from one system to the next. My latest incarnation feels right, but I've said that before. I always experience a temporary boost in productivity, but I really think that's just because I've spent time reviewing and prioritizing all my "stuff." Still, I'm only cautiously optimistic about my new system.

So, what am I using now? Google notebook. I'm basically following these instructions, except that I'm not using a Next Actions notebook with a section for each context (@computer, etc.). Those instructions were written before notebook had tags, so I'm leaving the actions with the projects and tagging (labeling in google-speak) them with contexts. This avoids the problem of losing the tie between projects and next actions that lots of people (like myself) find troublesome.

I can also now keep a link to mail. I've always lamented the fact that gmail didn't have a single URL per message. Well, now apparently it does! Those guys must have updated the software. Now I can use the notebook firefox extension to select the text in an email and create a new note, which automatically creates a link to the email. Unfortunately, outlook web access doesn't have a unique URL for each message. You can open as a web page and that kind of does it, but then you don't see all the messages meta-data and if you move it to a new folder, the link breaks. A long time ago, I tried notebook-like application called evernote, which could link to outlook mail, but it was a desktop app with a local file that wasn't portable unless I carried a thumb drive around. (Of course now evernote is working on a web version.)

In violation of GTD principles, I've added @today and @thisweek tags. This way if I lose focus on just what I *need* to be doing right now, I click on those labels. In the week I've been using this, I've found this very convenient. "Today" and "this week" covers my most common short term horizons. When I do a weekly review (probably Friday afternoon or Monday morning), I can label things with "@thisweek" as I come across them. (Maybe I should add a @thisFY -- fiscal year?)

What do I do when I finish a task? At first, I just deleted them, which didn't really bother me too much, but then I had a better idea. I created a new "Journal" notebook. I make a new section for each day, 2008.03.06, 2008.03.07, etc. Then, when I complete a task, I select "Move" and select "Journal" and section "2008.03.07." That way I know what I did today. Pretty neat, huh?

So far my annoyances with this system are:

Keyboard shortcuts. Geez, are there any? I can't find any documentation although I accidentally figured out that hitting the right arrow twice at the end of a note begins a new one.
Not being able to create links with my work email quickly. I usually just annotate "from:Fred sent:3/06/2008" which I can cut and paste into windows desktop search and find it in a jiffy, but it's not ideal. I could forward to gmail, but that has its own issues, and it takes several minutes for the forwarded mail to arrive.
Google notebook's mobile version doesn't support labels. One workaround here is to put my mobile contexts (like @errand) into a next actions notebook like he does.

It's been a while since I've posted here

2008-03-01T07:18:00.001-08:00

I should probably start doing more learning and less running. It's been over a year since I've put anything hear.

My running log project has been languishing. When last I left it, I had decided I would use pygtk, libglade, matplotlib, numpy, and sqlobject. This is a nice solid, pretty mature base for such a project. Then, of course, I'm tempted to use the enthought tools, but it seems to be a moving target and documentation can be tough to come by. I'm tempted though because it looks like "cool stuff" and coming up with a project to force myself to learn it might be worthwhile. Finally, my unfocused brain is thinking about implementing it as a web application using turbogears in order to learn a different set of "cool stuff."

What's new?

2007-03-11T14:10:00.000-07:00

Picked up Zen and the Art of Motorcyle Maintenance off my shelf and started reading the first chapter. I found one of my favorite quotes: "What's new?" is an interesting and broadening eternal question, but one which, if pursued exclusively, results only in an endless parade of trivia and fashion, the silt of tomorrow.

I started playing with the HRM program again yesterday and made a little progress on displaying the log summary data. I also changed the user interface arrangement to match more of an email look and feel.

kiwi

2007-01-26T22:39:00.000-08:00

After playing around with kiwi today, I decided to put it aside for awhile. It looks promising, but I started to get the feeling I was off on a tangent. One thing I liked was that it didn't seem as insistent as traits was on what I use for a data model.

form validation

2007-01-24T07:06:00.000-08:00

One of the nice things about Traits UI was that once you described the data, the validation of user input took care of itself. In my new Traits UI-less world, I'll have to find another way to do that. Being lazy, I googled around for existing libraries. Formencode does this for html and isn't tied to html, but I haven't found examples using pygtk. It's also supposed to be used by sqlobject? Maybe I can rely on sqlobject exceptions to do the work for me? Another possibility is kiwi, which actually serves as an MVC framework of sorts for pygtk. I will say I'm loathe to include more dependencies, but kiwi doesn't scare me that much.

splits -- some progress

2007-01-23T20:06:00.000-08:00

I'm writing splits from python, pygtk/libglade, matplotlib, and sqlobject/pysqlite2. I've written the code to pull the data from STraM, which uses an access database and a set of files (for storing the R-R interval data), using odbc from the pywin32 extensions. The data is then stored in an sqlite3 database, which has the very nice property of being binary compatible across platforms. The would-be linux user (like myself) can install the windows app, import the data, and see how he likes it on windows. Then, if he's so inclined, he can start using the program on linux.

He can do all of this if I finish writing it that is :-) Here's something at least.
OK, it's not much, but I figured out how to do listviews in gtk and embed a matplotlib graph. On the backend, I've got the persistence piece figured out, which was a stumbling block before.

version control and plain text is good

2007-01-23T19:22:00.000-08:00

I started using subversion for a few coding projects, keeping the repository on my web host, so I can "svn update" and "svn checkout" from wherever. Very nice. Why can't life be this nice for projects other than software? How about that report your working on? Well, the problem is binary formats. How do you handle conflict resolution? I'm sure if you're working on a word document there's some method for conflict resoluton, but I'm pretty sure it involves some big, complicated tool. Maybe I'm just living in the past, but it seems like things were easier when I wrote papers in LaTeX and they looked great. Can life still be lived in plain text?

more implementation, less invention

2007-01-03T15:19:00.000-08:00

Ironically enough, an email to the enthought list with a link to this article convinced me to just use python-glade, matplotlib, and sqlite. These are all known quantities and I can easily see a straight line from here to the finish with them. It might be less elegant, require more code, and be less easy to change in the future, but at least I can see the path!

glade pygtk matplotlib tutorial

2007-01-02T12:45:00.000-08:00

I read a tutorial for embedding matplotlib in a pygtk application! Cool, now I can expand my "hello, world" python-glade playing with "hello graph!"

A project has a plan

2007-01-02T12:02:00.000-08:00

I don't know why this thought struck me as I was re-reading http://www.emacswiki.org/cgi-bin/wiki/PlannerModeCommentary, but it did.

Right now, I have project pages on my wiki with sections entitled "Next Actions," "Projects," "Someday/Maybe," and "Notes." Projects and sub-projects are where sub-projects go. The planning sort of goes on in the "Projects" section although it isn't explicit.

Maybe each project needs a plan with required sections (could be a just bullet instead of a section).

"First, create the idea, what you want to achieve." A section called "idea" or maybe "Desired outcome" could spell this out.
"...step by step, identify every barrier to the realization of your idea, and devise a countermeasure to overcome it." This is similar to the "obstacles" section of David Pollard's GTD system. Makes sense.
"Third is to break the stages of the plan into parts that are not overwhelming in their complexity. It is at during this phase that a plan is turned into task items, each to be accomplished within the span of one day’s time. If a task requires several days, break it up further. The smaller it is, the less your mind will recoil from attempting it." It seems I have this in "Next Actions" but should each sub-project have its own page with its own next actions? I don't think so, but that just might be recoiling because it wouldn't be easy with my current tool.
"Fourth is to monitor your progress, identifying problems and correcting for them as you go. Some plans start out unachievable, and remain that way indefinitely, due to a simple lack of observation. If nothing is working for you, change it. Otherwise, your plan is merely a well-crafted wish." In GTD-land, this is the weekly review. Doing this regularly is on my todo list!

use traits/traits UI only from enthought tool suite?

2007-01-02T09:49:00.000-08:00

I played with python-glade today thinking maybe I'd just try things the old-fashioned way. I started to think about it though and while getting a data viewer up and running wouldn't be hard, I'd have to write code to validate the inputs when somebody wanted to add a log entry (enter distance, etc.). That sounds painful, so maybe I'll give traits a shot.

Splits -- should I build it with the enthought tools?

2007-01-01T16:36:00.000-08:00

I tentatively named my little STraM replacement "Splits." (Suunto Training Manager -- aka STraM is the software the T6 uses.) Now what should I build it from? I decided use python because that's what I'm familiar with and I think it's quite nice. I figured I'd need a UI, a graphing library, and a database that have python bindings.

My initial inclination was to use wxpython, matplotlib, and pysqlite2, but the enthought tool suite was kind of intriguing and without much work I created some model classes and views and then did the magic "splits.configure_traits()" and got a nice gui I could use to browse logs like so

So far I've just used traits and don't know quite where to go from there. There's a lot of tantalizing "stuff" in the enthought tool suite but not much in the way of documentation (or an indication of whether it works).

I need some method for storing the data -- "persistence" for those with computer science degrees. I see enthought.persistence.state_pickler.py, enthought.persistance.versioned_unpickler.py, enthought.sweet_pickle. Is one of these good enough for me to entrust someones sacred running logs with? I'm pretty confident in sqlite, but I don't know about pickling.
I made the simple graphs in the screenshot just using Plot from enthought.model.numeric_ui which I happened upon somehow. How configurable is this? Should I use chaco? chaco2? pyface with matplotlib?
What other tools are in the enthought tool suite I can use? I see enthought.units. Maybe this will help me not reinvent the wheel in handling mph, minutes/mile, km/h, etc.
Should I be using this envisage framework? What is this? What would it buy me? Is there any documentation? As far as I can tell the best way to understand it is for me to look at eclipse, a java framework on which envisage is based.

Lots of questions. Figuring out the way I'm going to store data is really the next step.

Getting back to linux

2006-12-31T09:49:00.000-08:00

I'm trying to make the move back to linux. I used linux as my home desktop from October 1993 to January of 2006 before the accumulation of windows-only software and gadgets (running watch, digital video camera, etc.) had me giving in. Since then, I've been trying to figure out how to get back to linux one program at a time. To that end I've been toying with the idea of writing a cross-platform replacement for the software my Suunto T6 uses. I've been playing with the enthought tool suite on windows and have made some progress. Now I'm to get what I have working on windows working on linux.

About a month ago, I bought a cast off computer from work and a KVM switch so I could have a linux box sitting next to my windows box. With the holidays, I finally got a chance to mess with it. Trying to install Ubuntu 6.10 and get it set up, I hit a few bumps. The basic Ubuntu installs beautifully. The first headache came when I tried to get flash, etc. installed to "desktopify" my new box. There seemed to be "more than one way to do it" but I ended up using something called automatix2 and was on my way.

Next on the list was installing the packages that I need to do an in-place build of the enthought tool suite. Scipy was a problem. I installed python-scipy and the myriad of stuff it wanted to install, but the build died right away because it couldn't import scipy_distutils. So I tried to install scipy-core which provides scipy_distutils. Unfortunately python-scipy-core conflicts with python-scipy even though it recommends python-scipy. Hmm. Well, python-scipy includes numpy.distutils instead of scipy_distutils so I built with numpy and it seemed to compile OK (some warning, but nothing alarming). Running the testing/marathon.py though had lots of failed tests.