Lucid dreaming

Recently, at a party, I got into a conversation about lucid dreaming. A lucid dream is a dream in which you know you're dreaming.

I first became fascinated by this subject when I started to wonder why you always know that you're not dreaming when you aren't but why you don't when you are. Then I actually experienced a dream where I realized that I was dreaming and that really spooked me.

Why aren't all dreams lucid?

I didn't feel like I was dreaming when I was dreaming. Maybe I'm doing something equivalent to dreaming all the time - spending my life in a daze and not really knowing it. I decided that I was going to work on making all my dreams lucid dreams.

Talking to the guy at the party I was reminded that many people don't remember their dreams at all. Other have never had a lucid dream. Others don't believe it's possible.

Yeah, lucid dreaming is possible. It's not as fun as you might think but it's possible.

After practicing for a few years I managed to have lucid dreams about one every other night.

Many people seem to think that if they knew they were dreaming they'd be able to do all sorts of stuff they've always wanted to do. In practice, my experience is the dream already has an agenda and if you don't play along you wake up. That's not to say you can't do some cool things in dreams, it's just a lucid dream isn't your own personal holodeck.

Dreams are manifestations of your expectations.

Let's say you built a computer. This computer makes up models of the world. It tries to find patterns and guess what's going to happen. This computer is very good at it and after some amount of training develops a whole bunch of models of how things work. It knows when lunch is. It knows you like coffee at the start of the day. It knows if you let go of something in mid-air it falls. It's got a fairly good view of the world.

Now you want to take it to the next level. You want to make this computer interact with the world very quickly; without having to think about what's going to happen. You essentially want it to figure out that if I drop something fragile it should attempt to catch it! Fast! There's no time to think about what's going to happen when you let go then figure out it's going to fall then figure out that it's fragile then figure out that it will break and that you will get mad and yell at it. It needs to know the rule that: If fragile item falls you (the computer) catch it.

How do you do that?

Well, as I said, the computer knows about how the world works. So if you bring up its memory of the lab and then ask it: "What sort of things could happen in the lab?" it will know: the same few people tend to walk in, in the morning. Lots of these people have coffee. Coffee is always in cups. The sun shines through the window in the morning. People occasional brush past things and push things onto the floor. etc.. blah blah.

Really, you don't need to teach the computer about the world, you can simply run it's models of the world against itself and it will figure out (eventually) most of what it needs to know. So long as you take more or less random branches out of the possible scenarios you'll end up producing experiences that have never happened. This is useful to train against.

The human brain is a bit like this. It makes models of the world and builds up expectations. Dreams just run these expectations against one another... presumably to help you react better if this situation should come up in real life.

Actually, there's another important factor: emotion.

If dreams were only about expectations you would have really boring dreams. Most of the stuff we expect to happen is fairly dull. The thing with dreams is they often choose which expectation wins based on emotional response.

Here's an example. Let's say you were in high school and you're standing in the cafeteria. Consider these events:
- Someone you're friends with waves at you from across the crowd.
- Someone drops their tray.
- There's a good seat available for you a few feet away.

Not much emotional response. How about these:
- You buy some lunch but discover that your pocket has a hole in it and all the money is gone!
- All the seats are somehow taken.
- A teacher walks up an casually mentions you just got an A on a test you were worried about.
- Terrorists!

Dreams go for the big emotional responses.

If you become lucid you can't suddenly start trying to make things happen that are just "neat". "neat" doesn't play well in dreams. "Dangerous" is fine. "Angry" is fine. "Really totally awsome" is fine.. "neat" isn't.

.. and don't try things just to find out what will happen. It's a dream! Trying stuff in a dream just to see what will happen is like having the following conversation with yourself:

"So.. what will happen if I do this?"
"I dunno. What do you think?"
"No, I'm asking you."
"Well, what do you think the answer should be?"
"Actually, I'm just curious about what you think should happen."
"Well what do you think should happen?"

If you do this enough, you dream will go "ah, screw this" and wake you up.

In a dream your job is to react to things not make stuff happen. You can make stuff happen but you have to go along with the flow of the dream. Try to make it a reaction to something.

Oh no a tiger! Good thing I'm in a dream since that means I can fly away! (model of how thing works goes "yeah ok".. so you fly.. Good thing your instinctual brain never took physics).

That's about the long and short of it.

In the end I started having weird semi-lucid dreams. In this semi-lucid dreams I know on some level that I'm in a dream because I know that all the things I used to do in lucid dreams will work.. I also know I can just escape by waking up. Well, unless I have a really bad nightmare in which case I an usually so focused on the moment I don't get that strange "this place follows dream logic" feeling. What I think happened is my behavior adapted to included the fact that, in dreams, the rules are different.

Sometimes I think my brain has built its own parallel universe. I have a "dream montreal" that doesn't look like real montreal but is more-or-less consistent every night. I have a map of my home town that includes things which are completely wrong but very consistent between dreams. They aren't spatially correct but they are emotionally correct. That is, the distance between two buildings is a "short" or "long" walk. That the main street contains an "annoying" amount of traffic. That buildings in the new art of town are "taller" and "newer" than the old part. That some buildings are "tall" while others are "scary tall". It's hard to describe. It's not montreal. It's obviously not montreal.. but it feels like it.

.. oh and all the elevators are dangerous and unreliable. I know that if the elevator doesn't work it's because I'm dreaming :-)..

So how do you get into a lucid dreaming state?
Read this link:
http://www.lucidity.com/LucidDreamingFAQ2.html#techniques

The most important thing is to remember that you can never tell you're dreaming! Especially if you've woken up from a dream! Your body doesn't like waking you up for no reason so it will tend to fake waking you up. When waking up from lucid dreaming always make sure you're not still dreaming.

.. checking for dream signs requires excellent self control.. Actually I think the ability to extract yourself from the moment is the primary benefits to pursuing lucid dreams.

So read all that stuff and you will have a lucid dream tonight. Seriously. (Go flying in reaction to something. It's scary fun.).. good luck having a second lucid dream but the first one usually comes easy. :-)

good book:
The Head Trip - It deals with various states of consciousness including lucid dreaming. No LSD or drugs here. All these states are natural ones.

If you like lucid dreaming you might like to experience the watch. Another fun altered state of consciousness easily accessible by messing with your sleep patterns. :-)

Why UI design applies to APIs and frameworks

That blasted book! (see previous post).. Now it's got me thinking about API and framework design..

Ok.. So I'm reading a chapter about David Heinemeier of 37signals. He's the dude who wrote ruby on rails. In it he says the following:

"You need to innovate on behalf of your customers, but they don't often know what they want. And it's the same thing for programmers. If you went around and asked them what they wanted in a framework, you wouldn't get a good product out of that. You need to be able to source input from alot of sources, and then have your vision of what it's going to be and then drive that."

(emphasis mine)

There's this idea floating around that the reason a software sucks is because the people make it don't pay listen to the customer. All the layers of bureaucracy between the programmer and the user causes a disconnect. If we could just talk to the customer directly we'd be able to ask them what they want directly and just program that.

This is partly true but there's a problem: users don't know what they want.

I have a feeling that many who have not actually done a software project with real users see this as an arrogant statement. It's not.

Users are not programmers. They don't know interaction design. They probably don't have very good aesthetic tastes. If you let your users build your specification for you, you'll end up hearing the dreaded "This is what I asked for but not what I want". This is why extreme programming, which advocates the above, has a short iteration time.

On the subject of framework design.. Every once in a while someone will post a bit about how painful it is to do something or other in Java vs Python. Most of the time it's something trivial that's not really a Java-as-a-languge issue but Java-as-an-API issue.

Java's API can be irritating at times. My favorite example of this is reading a file.

Let's say you were in Python and wanted to read a file as text. Here's what your code would look like ->

   with open("hello.txt") as f:
     print f.read()

Now for the Java version. Settle in because this is a bit long.

First off let's do this as a function that returns a String

    public static String readFile(String fileNameToRead) {
        return null; //for now.
    }

Ok, so I didn't bother doing this for python. Beer with me here.. Mmm beer..

It's nice to get a quick poke about how verbose declaring a function is :-) .

Now We have a function let's create a file object:

    public static String readFile(String fileNameToRead) {
        File file = new File(fileNameToRead);
        return null; //for now.
    }

Unfortunately, the File object in Java doesn't have any method to read it's contents. To actually read a file we have to create an FileInputStream. Let's do that.

    public static String readFile(String fileNameToRead) throws FileNotFoundException {
        File file = new File(fileNameToRead);
        FileInputStream in = new FileInputStream(file);
        return null; // for now.
    }

The problem with InputStreams in general is that 1) You can't get a String out of them and 2) if you ask it to fill and array of bytes it actually won't fill the array of bytes. It might fill the array to completion but it probably won't. It's your responsibility to loop over the "read" method in FileInputStream an accumulate all the bytes you want.

To heck with that! DataInputStream can read a full byte array so let's use that!

ByteInputStream can wrap ANY InputStream and add functionality to it. This means you can use it to read bytes from a network connection, file on disk or anything else. It's very cute.

    public static String readFile(String fileNameToRead) throws FileNotFoundException {
        File file = new File(fileNameToRead);
        FileInputStream in = new FileInputStream(file);
        DataInputStream dataIn = new DataInputStream(in);
        return null; // for now.
    }

Notice that we had to add the "throws FileNotFoundException" to the end of the method there. This is because we're doing IO which could fail. If it fails it throw an exception. Java won't let you compile until you've told it how you want the exception handled. Most languages don't do this and just let the exception bubble up and kill the program. We're just going to tell java to throw it up to the caller and let someone else take care of it.

Ok, now we have a DataOutputStream but we can't read a String using it. It actually has a method called "readUTF()" which returns a String but it doesn't do what we want because readUTF() expects the data in the file to be in specific format. That format is only written by the corresponding method from DataOutStream.writeUTF()..

Ok, lets create the byte buffer!

    public static String readFile(String fileNameToRead) throws FileNotFoundException {
        File file = new File(fileNameToRead);
        FileInputStream in = new FileInputStream(file);
        DataInputStream dataIn = new DataInputStream(in);
        
        byte[] bytes = new byte[(int) file.length()];
        
        return null; // for now.
    }

Now we need to read in the bytes.. which is actually fairly easy now that we have our DataInputStream..

    public static String readFile(String fileNameToRead) throws IOException {
        File file = new File(fileNameToRead);
        FileInputStream in = new FileInputStream(file);
        DataInputStream dataIn = new DataInputStream(in);
        
        byte[] bytes = new byte[(int) file.length()];
        dataIn.readFully(bytes);
        
        
        return null; // for now.
    }

Notice the exption changed. This is because DataInputStream.readFully() throws its own exceptions. Since both exceptions are the same kind (IOException) we're just going to tell java that all IOExceptions should be dealt with by the caller.

Yay we have the file's bytes! Now bytes aren't a String so we still need to convert to a String...

    public static String readFile(String fileNameToRead) throws IOException {
        File file = new File(fileNameToRead);
        FileInputStream in = new FileInputStream(file);
        DataInputStream dataIn = new DataInputStream(in);
        
        byte[] bytes = new byte[(int) file.length()];
        dataIn.readFully(bytes);
        return new String(bytes);
    }

But wait, there's more! We still have to close the file! In the Python code this is done by the "with" statement.. We have to do this by doing this ->

    public static String readFile(String fileNameToRead) throws IOException {
        DataInputStream dataIn = null;
        try {
            File file = new File(fileNameToRead);
            FileInputStream in = new FileInputStream(file);
            dataIn = new DataInputStream(in);

            byte[] bytes = new byte[(int) file.length()];
            dataIn.readFully(bytes);
            return new String(bytes);
        } finally {
            try {
                dataIn.close();
            } catch (Exception ignore) {
            }
        }
    }

There we go. Now if anything goes wrong the file will be closed.

Wow... That's longer than the python version...

The thing is.. If this function existed in the API you could call it to print out the contents of a file by doing this ->

  System.out.println(readFile("hello"));

This line does the same thing as the python version. They could have put this function in the core API. They didn't.

I'm betting that what python has to do to provide that functionality is very close to what we did here.

Java-as-an-API doesn't have a very good file-reading / string-manipulation toolbox. This is an API issue. It can still do everything but it's a huge pain. Python rocks for this.

On the other hand Java-as-an-API has a massive framework for making GUIs. Python-as-an-API doesn't.. You need to use a third party library.

Just one more thing before I sum all this up: Java both as a language and as a toolkit was the totally wrong tool for doing applets.

People wanted to make banner and cute animated graphics and such.. Look at what people are suing flash for today and that's what people wanted to do with java applets.

Java applets should have had an API for doing animation. It should have had an animation studio attached to it.

... oh yeah and it had to have drop dead easy deployment..

... oh yeah and it shouldn't have cause the web browser to freeze up solid for 30 second whenever it was used...

.. I could go on but this is all well known.

Sun didn't do their research. Don't be like Sun.

Designing frameworks is hard. You need to balance a host of things from flexibility to speed to ease of use to power and it just goes on. If you don't know who your building for and what they need you're doomed to failure. They can't tell you either because they havn't thought about the problem very deeply.

The process you need to use to build a good API is the same process as the one you need to use to build a good interface.

You need to know your UI basics either way.

ok, I'm done.

Skating in the park with flounders

Wee, I just got back from playing hockey in lafontain park in Montreal.. It's about 0 degrees outside today so it's perfect weather for skating.. Not cold enough that you start to worry about the consequences of breathing.. not warm enough to melt the ice. Excellent.

heh heh.. While trying to stop someone from scoring I tripped and fell into the net.. I scored myself.. Luckily I wasn't hurt. I realized that it was about to happen so I put my hands out in front of me just as I fell. Those nets are heavy and you can seriously hurt yourself it you bang your head against it. We don't play with equipment either so you got to be careful.


I'm still trying to finish reading "Founders at Work" which is a great book that interviews the founders of major tech companies about what it was like to start up a company. These guys are crazy. They work crazy hours on practically no sleep.

anyways... I'm having trouble finishing the book because after a few pages I want to throw the book aside and go found a startup company. It's just silly.. I'll be sat there reading then I'll stop and wander around the apartment thinking about what sort of thing I'd like to invent and how I'd market it etc.. Eventually I'd calm down and sit back down with the book where the cycle would begin again.. It makes reading the book very slow going.

Anyway, I'm going to make some hot chocolate, sit down by the fire for an hour or two and read.. about 5 or 6 page.. thereabouts.. yeah.. bye.

War is not cheap.

Currently the total cost of the war in Iraq is about 500 billion dollars. I've often wondered what this war is worth in terms of opportunity costs.

http://www.nationalpriorities.org/costofwar_home

2.5 magnetic levitation /vacuum trains from London to New York. These puppies can get you from London to New York in 1 hour. They fly through tubes under the ocean that are almost at a total vacuum at 4000 mph on magnetic tracks. Much faster than flying.

http://www.klanrur.co.yu/forum/index.php?showtopic=45349

Pfff, in fact all of the US could be setup with coast to coast maglev links

http://en.wikipedia.org/wiki/Maglev_train#United_States

21 Manhattan projects

http://en.wikipedia.org/wiki/Manhattan_Project

1 600 $ tax rebate for every human alive in the US. That would stimulate the economy.

http://www.census.gov/main/www/popclock.html

35 Superconducting Super Colliders.. Physics would love you!

http://en.wikipedia.org/wiki/Superconducting_Super_Collider

34 ITER nuclear fusion reactors.. Wow.. that would be cool.

[This link is missing because X-Windows copy and paste is unreliable and I don't want to go find the stupid page again]

1 man on Mars. awww... looks like Bush wanted a war instead...

http://query.nytimes.com/gst/fullpage.html?res=950DE3D6163FF932A15752C1A96F948260

4 International Space Stations

http://en.wikipedia.org/wiki/International_Space_Station

340 Montreal Olympic Stadiums! (wow, that beast really was expensive)

http://en.wikipedia.org/wiki/Olympic_Stadium_(Montreal)

$3,749 per iraqi

http://zfacts.com/p/447.html

agh it goes on.. Carbon credits, electric car research, laptops for developing countries, UN back-dues...!

What's your favorite, expensive megaproject?

beep

Hello, I can't come to the phone right now. If you'd leave me a message I'd get back to you as soon as possible.

Hi there. Umm.. It's Andrew. You know, I hate answering machines. They defeat the purpose of calling someone. I mean I might as well email or leave a message on any one of the billions of message-board style communication mediums. Frankly those are better because I'm not put in the unusual situation of having to come up with something without a backspace key.. Also, I've heard what I sound like on tape. I wouldn't wish that voice on anyone. This has been a long message. I'm surprised I haven't been cut off. Some of those cheap answering machines have so little space on them that you can barely get a message out. Have you noticed you always get cut off at a point that makes it sound like you're insane or insulting:

"Hi there, well, I've been thinking and I'm come to the conclusion that you're just as much an asshole as.." beep!

"Hi there, It's me again, I was about to say just as much an asshole as I've been so I really shouldn't be mad.. Ok, well if you.." beep!

agh!

Try calling back.. nope.. tape full. ha! You'll have to learn to talk in sound bites.

So, you known what. I'm not going to tell you why I've called. I'm going to leave you my number and get you on the phone so I can talk to you. I've got more to ask you from you than a question. I want more from you than a reply. So call me back. My number is

The fuzz point.

When trying to predict something, it is often the case that you need progressively more processing power as you increase the precision of your predictions. There's a point, however, at which increasing the amount of effort you put into creating the perfect prediction runs into the hard truth that either your model or your initial reading of the system's initial state limit your prediction's accuracy. Past this point, there's no reason to invest more effort because the amount of precision you'll have in your answer is more than the margin of error for that answer. This is the fuzz point. It is the point of diminishing returns.

It can show up in interesting places. My favorite place is the classic, intractable argument over aesthetics. If you just work at a little more it would look better. This isn't always the case. Consider your ability to predict what people find aesthetically pleasing. Consider any data you have on the topic and how much error there's likely to be. Consider the amount of time you've spent arguing about whether the arrow should be green or blue. You've past the fuzz point.

Aesthetics aren't unique. A special case of the fuzz point shows up when prioritizing. Bug fixes and feature in software.

How accurately can you predict how long something is going to take to fix?
How accurate can you be in predicting how important a feature is to implement?
How long are you going to argue about it?

The motto is, past the fuzz point, flipping a coin is actually cheaper in the long run.

The fuzz point is very small for small bug fixes. So small, in fact, that you get multiple different sorts of penalties.

In many shops all bug reports must be prioritized before it's decided whether or not they are worth doing. For bug fixes < 4 hours weird things start to happen.

The cost of figuring out if the bug is severe becomes more important.
The cost of tracking down the cause of the bug tends to be much more important.
The cost of the bureaucracy of fixing the problem becomes very important.
The cost of merely context switching away from to bug for enough time for it to be prioritizes becomes important.
The difficulty in measuring the relative importance of all these things increases.

Small bug fixing is fuzz point land. If a bug takes a short amount of time, there's no point in prioritizing it. The amount of time you've spent just trying to figure out the true severity and the cause dominate. If the fix is quick, don't prioritize, do it now, on the main branch and the deal with the risk portion of the bug fix separately. (Essentially review the severity and risk of each bug and fix and decide if they must be back ported to the old branch for a bug fix.. Also decide if it's worth running it by QA. The answer is almost certainly yes.).

If you do this, however, you will notice that you're development will stop. This isn't good. The way of getting around this is to allocate a fixed amount of resources to the task and prioritize bug fixing in its entirety with the adding of new features.

If you must prioritize fixes then poll the list of bugs looking for important ones. Don't force everything to be run through the bureaucracy before anyone can get a time budget for it.) .

If you implement this make sure you clearly say how long to spend in the various stages of bug tracking before giving up (how much time trying to ascertain the severity vs how much time investigating for each level of severity vs how much time trying to implement the fix). This is a heuristic but it works fairly well, because bug fixes show up in timesheets so you can see violations.

Must be going. I've been spending too long on this issue.

HAPPY NEW YEAR!

I would like to wish all my friends a happy new year.. but I can't. I can't send them an email because that would be SPAM.. and I can't send them each an individual email because that would be an easily automated job and it's against computer programmer ethics to not be lazy in situations like this. I can't say HAPPY NEW YEAR! in my blog because only two people read it and I really want to tell everyone. I can't put "Andrew is wishing everyone a happy new!" in my facebook status because no one actually reads those things anyways. They exist only for self amusement and occasionally as an existential venting mechanism as in "Andrew is a spoon". Hummm.. I can't think of any way of doing it so I'll stick it in all these places and send personal notes to a few people arbitrarily. It's not perfect but it's the only way to avoid annoying myself.

Andrew is a fork.