December 18, 2007

Transitions and Epiphanies.

It's been a crazy couple of weeks for me...

First off, Lucas Newman is leaving Delicious Monster -- as of January 1 he will be an iPhone engineer. This is an amazing opportunity for him, one I would never ask a friend to pass up. We remain buddies, although I'm running around Zoka these last couple weeks telling every girl I see that Lucas was secretly super-hot for her and is leaving now, which I think is starting to annoy him. Although, honestly, they'll probably all end up throwing themselves at him and he'll end up on top, again.

For those keeping score at home, this makes Mike Matas, Scott Maier, Tim Omernick, Drew Hamlin, and Lucas Newman that Apple has hired out of my employ. Yes, in fact, 100% of Delicious Monster's ex-employees are now working for Apple! You'd almost think Apple would start to pay me to train people for them. Oh, well. It's every kid's dream to work there, I can't say I blame them. Heck, I might work for Apple myself if they ever asked. And, like, wanted to give me EIGHTY ZILLION DOLLARS.

Also, seriously, if you want to work for Apple, you MIGHT want to, you know... GET TO KNOW ME.


Mike Lee is staying at Delicious Monster -- for now... DUM DUM DUM! You have to figure he's playing the various Apple teams off each other -- when you work at Delicious Monster, you don't jump for the girl that asks you to dance. Mike's all: "CoreAudio? Don't waste my time, sweetheart." "OS X Server? I'm sorry, you're not even getting an interview." "Ali Ozer and Scott Forstall got into a fistfight over me at lunch today? Now, see, these guys understand what kind of ball we are playing."


I realized tonight that I had yet another problem with CoreData, and it was a doozy, and not something where I could just put a hack on it. In fact, it was indicative of a fundamental architecture mismatch that I've been struggling with since I started this project.

So, this is a little vague, but I thought it might be important to document the process. Basically, when I bang up against a wall, I start looking bigger and bigger and bigger. Like, imagine I'm having trouble with a crumbling wall in an aqueduct -- my programmers brain does this: "Ok, why did I build this wall?" To keep the water in. "Why do I have water?" Because you need that to turn the water-wheel. "Is there some other way to turn it?" Not easily. "Why must it turn?" To power the grinder. "What needs grinding?" Corn. "Is there some other way to grind it?"

I'll get to truly huge things, where I start asking if the world even needs an app that catalogs books and DVDs and now boardgames when we could all be under five feet of water in a few years. Then it's time to take a nap and wake up and start again.

But my point is, you HAVE to question all the basic assumptions that led you to where you are, or you end up spending all your time writing the wrong code. I have always said that if you give me a perfectly spec'ed out program (one with a spec that can actually work, that I'm not going to have to modify as I go along), I can write that program for you in days. Always. The problem with coding is (a) fighting with frameworks, and (b) trying to figure out how the program should look, work, and interact even as we code it.

So we end up spending a lot of times fixing bugs in code that we really shouldn't have written in the first place -- code that doesn't really help the user, that just makes the app more complex, that is for a feature that never should have been put in, or is interacting with the user incorrectly and we're just putting spackle on a wall that's crumbling.

So here I am, tonight, running into my 1,000th bug with the fundamental mis-architecture in CoreData, which is that interacts with the UI layer and the disk layer / undo layer all using the same mechanism. They all rely on -didChangeValueForKey:, which is a huge mistake, because it means that, as a programmer, I can't sneak any data in -- I can't change a value without it creating an undo event.

Consider if, for example, I had a clock and its hands were CoreData objects. As they move forward through time, their position updates, so I'd tell them to update. And each time I did, an undo event would get pushed -- so the user actually could undo time.

This is obviously a contrived example, but it also points to the fundamental problem -- CoreData objects can't mix undoable and non-undoable changes.

So I've been struggling for three years now, trying to bend and hack and cajole CoreData's undo architecture into allowing me to do some actions synchronously and some asynchronously. (For instance, obviously, once the program has downloaded a cover from Amazon in a background thread, you don't want to UNDO the download -- it's not actually a state change, it's just a cache change -- yet, by default we end up with an undo event on the stack, in the MIDDLE of whatever the user is actually doing in the foreground.)

Fight fight gnash gnash complain complain. Tonight I hit on it. I needed to step back. Why isn't this working? Because undo wasn't designed this way in CoreData.

Well, I have undo in Delicious Library 1. It's not "magic" like with CoreData, but it works. In fact, now that I am thinking about it -- I've spent months and hundreds of lines of code trying to get CoreData's "magic" undo to work, when, in fact, there are really only FOUR actions that are ever undone:

1) Add a book -- undo to delete it
2) Delete a book -- undo to add it back
3) Change a property on a book, like its title or author -- undo to change it back
4) Make a loan -- undo to return the book
5) Return a book -- undo to re-make the loan


There's only five damn methods, at the top level, that need to participate in undo. It's pretty obvious I should be managing my OWN undoManager, turn off the one in CoreData, and just use CoreData for what it is EXTREMELY good at, which is minimal change tracking and fetching and storing data VERY VERY quickly.

Suddenly all these issues I've been having disappear. I don't have strange extra undo events on my stack when I fault in an object, because although CoreData might think my object changed, it's not driving the undo manager any more -- and when it goes to save, it's going to quickly discover there's no real substantive changes and just discard the whole event.

I don't have to try to work around some undo events by turning undo on and off, which required me to flush CoreData's transactions queue by hand, which was extremely sketchy because if you do it in some circumstances (eg, the middle of inserting a new object) the object will be corrupted.


I haven't started this yet -- I'll try it tomorrow. It's nice -- it'll pick up a bunch of the remaining issues I'm having in DL2, and should give us a good solid beta. The important thing here is, I was just too married to part of the code. I was so into using CoreData's magic undo that I kept going farther and farther to make it work, when I really needed to say, "Ok, this doesn't work in this situation, I'm doing my own undo in 40 lines of code."

Labels: , ,


Blogger Wil Shipley said...

Mike Lee wrote a touching farewell to Lucas on his eponymous blog at Atomic Wang.

I guess I'm not as personally heartbroken as Mike is, because I see this as the end of me being Lucas' boss and the beginning of Lucas being only a friend, and I'm a better friend than I am boss any day.


December 19, 2007 6:00 PM

Anonymous Robert Clair said...

I have no experience with CoreData, but using bindings has the same sorts of problems - it doesn't work very well with undo (see all the threads in the Cocoa archives). The usual answers are "put the undo registration in the set and get methods" (unaesthetic - your model knows about the undoManager) or "observe the variable with KVO and register the undo if it changes" - which can be a pain to set up and you have to worry about disentangling the observer and observed object if the observed object can go away.

But both of these suffer from the more general problem that they register undo's for any change to the variable - model initiated or user intiated. Not what is wanted in many cases. Much as the bank might like it, the user probably shouldn't be able to undo the change to the balance variable made when the banking program notices another month has elapsed and adds an interest payment. You could work around this with flag variables and stuff, but the code quickly gets uglier and more fragile than just doing everything from scratch yourself.

This is symptomatic of a more general problem with bindings: they write much of the controller layer for you - but in the form of an impenetrable blackbox. There just aren't enough hooks to get in and do custom things.

Between this and the fact that you can't step through the "magic" with the debugger if there is a problem I've abandonned the idea of using binding for nay but the simplest situations.

December 20, 2007 7:01 AM

Anonymous Bill Coleman said...

Steven Cover had a good analogy for this, in his book "Seven Habits for Highly-Effective People."

Basically, you could be working away, hard as you can, hacking your way through the jungle. Then, one day, you climb up in a tree, take a look around, and scream out: "Wrong Jungle!"

December 20, 2007 1:34 PM

Blogger Wil Shipley said...


I agree with your analysis of the problems inherent in undo, but I disagree that bindings makes them harder or doesn't work with them -- but, again, I thought the same things about CoreData until I realized that I should just eliminate CoreData from my undo decisions, and suddenly everything is working swimmingly.

I think I should make a whole blog post on undo, but I don't think I necessarily have the answers.


December 20, 2007 4:50 PM

Anonymous Doug Dickinson said...

oh dear, this touches a nerve: my Very First CoreData test project was...a clock! And exactly what you described as happening happens. But I was young and under the spell of the Apple Kool-Aid, so I thought, hmm must be me. But no, bitter experience has shown that, as you discuss, there's no simple way around it.

Before the advent of CoreData I had been archiving objects I wanted to save and unarchiving on open. This worked well and had the advantage that the same code I used for saving could be used for copy/paste. With CoreData I've had to implement extra stuff to put data on the pasteboard. I suppose archives have the disadvantage that all the data ends up in memory upon opening the file.

So, if I'm not using the SQL storage with a large dataset, and I have to do my own undoing, and I have to do my own copy/paste, what exactly am I gaining with CoreData?

And that's the point I'm at today. My first project to try out new stuff in Leopard is not using CoreData.

Now I'm running into that all-or-nothing-at-all approach Apple favour in using layers - _every_ operation is animated, unless you take special precautions. And those precautions (like pretty much everything else) are sketchily documented at best. I much prefer what they've done with views and windows where you do the operation un-animated to the object or animated to the proxy of the object. That's pretty minimal coding effort for maximal choice.

Doug --dd

December 21, 2007 12:47 AM

Blogger Wil Shipley said...


What you are getting with CoreData is the ability to scale, and the ability to load/save complex models without worrying a lot about the scale or the details of the data. For small datasets it's not going to be a huge win

For anything that can scale, it's from a factor of 100 to 1,000 times faster. Databases really are a huge win.

Just don't do your undo in CoreData.

I'd love it if CoreData gave us access to their mechanisms to archive (for copy/paste and for undo), but until then you can use something like:

[myManagedObject setValuesForKeysWithDictionary:[mediumDictionary dictionaryWithValuesForKeys:myManagedObject.entity.attributesByName.allKeys]];


December 21, 2007 12:53 AM

Anonymous Pierre Lebeaupin said...

Very interesting... You should file this in the Pimp My Code sidebar. At any rate it's enough food for thought while we wait for the next code insult session.

December 21, 2007 5:27 AM

Anonymous twobyte said...

I can't disagree more with Mr.Shipley. Core Data is great for undoing/redoing changes. You have to remember that undo provides grouping, so do read class description from time to time.

December 21, 2007 11:41 AM

Blogger Wil Shipley said...

Are you trolling me? Is this Mike? That's you, Mike, isn't it?

Seriously, I know about undo groupings. The problems have almost nothing to do with that.


December 21, 2007 12:28 PM

Anonymous Jim said... app that catalogs books and DVDs and now boardgames...

Wil, what was that about Delicious Library cataloging boardgames? I don't see a boardgame medium on the "add entry by hand" widget. (DL version 1.6.6) Is this an upcoming feature, or am I missing something?

January 10, 2008 7:25 AM

Blogger Wil Shipley said...

Boardgames are planned for version 2.0 -- but, of course, I make no promises until we've actually announced and/or shipped.


January 11, 2008 12:00 AM

Anonymous Brian said...

Thanks for the post Will. I have been struggling with a different undo problem. I can't get the information about what undo changed so i have to throw away my view objects and rebuild for every undo/redo. All the objects have calculated values, so i need to know what changed. You can observe MOs, but undo a MO to before it was created and all bets are off.

Anyway the answer is the same, do it myself.

February 06, 2008 9:10 PM

Blogger Kai said...

"So we end up spending a lot of times fixing bugs in code that we really shouldn't have written in the first place -- code that doesn't really help the user, that just makes the app more complex, that is for a feature that never should have been put in, or is interacting with the user incorrectly and we're just putting spackle on a wall that's crumbling."

Can I move to california and fetch coffee for you or something?!

I've spent...countless hours arguing and preaching this same sort of thing. It always ends in tears :)

March 12, 2008 8:03 PM

Blogger Wil Shipley said...

Can I move to california and fetch coffee for you or something?!

Well, I suppose you could, but it might not be the best use of resources, since I live in Seattle, and also I work out of a coffee shop and they bring it to my table anyhow.

But, thanks for the sentiment.


March 12, 2008 8:39 PM


Post a Comment

<< Home