NSIncrementalStore – The future of web services in iOS / Mac OS X

The Problem

You may be interested in how to wire up or sync CoreData to a remote web service.  There are plenty of frameworks to do this, like RestKit, RestfulCoreData, and various defunct libraries (CoreResource, etc.)  The problem with these libraries is twofold:

• They assume things about your backend (like “It’s REST!”  or “It has sane queries!”  or even “It parses!”) that you know are not going to be true of whatever backend you’re working with.  Especially dangerous are super chatty protocols that pull down more data than you need, which you know will change halfway into the project when everyone discovers that it’s slow, causing you to rewrite a bunch of fetches.  Over and over again.
• They require you to run data queries against some arbitrary Cocoa API, which then multiplexes to both CoreData and the backend.  This is bad because it’s not portable, because if the backend API changes, your code will have to change, and because it’s unclear to what extent CoreData features like faulting and proxy arrays “just work”, which you intend to rely on for performance.

As a result of these items, people end up rolling their own sad little sync engine that is just a bit slightly custom for each application, highly coupled to the backend and changes as it changes, and cannot be effectively re-used.  Or alternatively, they end up writing their own [MyAPI doSomeKindOfFetch] helper classes that require a lot of thinking in application code to use correctly (caching policies, etc.)

Instead of doing this, you should be using NSIncrementalStore.

NSIncrementalStore

NSIncrementalStore is perhaps the best-kept secret in iOS 5.  It doesn’t show up in any advertising materials, barely rates a passing mention in WWDC tech talks, and has the world’s shortest programming guide, at only a few pages.  It is so well hidden that I only discovered it very recently.  But it’s easily my favorite new iOS 5 feature.

Essentially you can now create a custom subclass of NSPersistentStore, so that instead of your NSFetchRequest hitting a local SQLite database, it runs a method you define that can do something arbitrary to return results (like make a network request).  This means that fetching remote objects is now as easy as

NSFetchRequest *myFetchRequest = [NSFetchRequest fetchRequestWithEntityName:@"MyEntityName"];
NSArray *results = [moc executeFetchRequest:myFetchRequest error:nil];

Even cooler, we now support network faulting. So a code snippet like this:

for(MyEntityName *entity in results)
{
    NSLog(@"%@",entity.someProperty); //potentially a network request
} 

(Of course, it doesn’t have to be. Your custom subclass can fulfill the request however the heck it wants to, including from a local cache.)

So why should you be using NSIncrementalStore? One reason is because it lets you, in application-land, use maximally expressive queries, independently of how bad the backend is today. Perhaps you are working with a terrible backend that only supports 1 API call that dumps the entire database.  You can still write powerful queries in your application to request highly-specific data.  Then you can respond to those application requests by serving up the highly-specific data from cache.

But when the backend guys finally wake up and realize it’s a bad idea and give you five different kinds of requests, you only have to make that change once.  The application is still emitting highly-specific queries.  You look at the query that the application wants you to serve and select the API call(s) that are most efficient to serve that query, doing whatever sort or filter that the backend doesn’t handle as an in-memory step.  And you only have to worry about this in one place.

And then as the backend slowly improves, you can slowly take advantage of more and more features.  As a slightly absurd example, if some day they let you execute raw SQL (the horror!) directly on the backend, you can write a little SQL emitter that takes CoreData queries and emits SQL that fulfills them.  The application continues to work at every step, it just gets incrementally faster as the network traffic gets more efficient.

And CoreData really is a fantastic API to use to talk to remote web services, because it lets you interact with remote objects as if they were local objects.  Everybody has forgotten–this was originally the problem that Objective-C / Cocoa was designed to solve!  We do not need yet another function call to retrieve a pansy flat representation of a record on a server.  ObjC has had a better solution than that for 30 years!

NSIncrementalStore – the unofficial guide

As cool as NSIncrementalStore is, it is missing a lot of love from the Apple documentation writers.  (Like any new API, there are undocumented bits, and some scary legitimate bugs, that nobody has gotten around to fixing yet.)  The best resources on it are

• The NSIncrementalStore and NSIncrementalStoreNode class reference
• The Incremental Store Programming Guide (which is unfortunately way too short)
• A brief treatment in this WWDC talk which is illuminating
• The various resources on NSAtomicStore (class references, programming guides) are also very helpful in filling in missing details.  NSAtomicStore is another type of subclassable CoreData persistent store which was introduced some time ago and is better documented.
• (and hopefully, this blog post)

Init

The first thing you have to do, of course, is override init in your NSIncrementalStore subclass.

- (id)initWithPersistentStoreCoordinator:(NSPersistentStoreCoordinator *)root configurationName:(NSString *)name URL:(NSURL *)url options:(NSDictionary *)options {
     if (self = [super initWithPersistentStoreCoordinator:root configurationName:name URL:url options:options]) {

    //set up caching, HTTPClients, or other ivars
    }
    return self;
}

Load metadata

When you chain to the super initializer, you will get a loadMetadata: callback.  Note, and this is very important: if you fail to set the appropriate metadata here, your incremental store will be improperly initialized.  This can result in strange EXC_BAD_ACCESS errors that I see people complaining about on devforums.apple.com.

- (BOOL)loadMetadata:(NSError *__autoreleasing *)error {
    [self setMetadata:[NSDictionary dictionaryWithObjectsAndKeys:@"mystoretype",NSStoreTypeKey,@"arbitrary-uuid",NSStoreUUIDKey, nil]];
    return YES;
}

You are required to set up a NSStoreTypeKey and an NSStoreUUIDKey, and if you fail to do so the incremental store will not be initialized.  See the appropriate documentation in NSPersistentStoreCoordinator to learn more about these keys.

Setting up NSPersistentStoreCoordinator

Now, elsewhere in your application, you set up the NSPersistentStoreCoordinator.  You tell it to use the custom NSIncrementalStore like this:

NSString *mycustomstoretype = @"mystoretype";
[NSPersistentStoreCoordinator registerStoreClass:[MyIncrementalStoreSubclass class] forStoreType:mycustomstoretype];
coordinator = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:MANAGED_OBJECT_MODEL_REF;
persistentStore = [coordinator addPersistentStoreWithType:mycustomstoretype configuration:nil URL:nil options:nil error:&err];

Easy as punch.  Notice on line 1, we set the string to “mystoretype”, which is the same string we used in our metadata.  Failing to make these strings match in my experience sometimes (but not always) causes the persistent store to fail to initialize, with an error message complaining that the strings don’t match.  So my advice to you would be to store that string in a common location (like in the .h file of the NSIncrementalStore) to ensure that they match.

Handling fetch requests

Now we have to actually handle the fetch requests.  This is where things get tricky, because this method

• Can handle fetch requests or save (e.g. write) requests
• Can handle fetch requests that are looking for an object, fetch requests that are looking for a count, or undefined future request types
• Does not actually return any of the object’s properties or attributes, which are requested on a different callback
• But is responsible for filtering or sorting on those properties

I’m going to show you just a dummy implementation to get you started, because the full scope of things you can handle in this method is extraordinary.

- (id)executeRequest:(NSPersistentStoreRequest *)request withContext:(NSManagedObjectContext *)context error:(NSError  **)error {
    if (request.requestType == NSFetchRequestType)
    {
      NSFetchRequest *fRequest = (NSFetchRequest*) request;
      if (fRequest.resultType==NSManagedObjectResultType) { //see the class reference for a discussion of the types
        if (fRequest.entityName isEqualToString:@"MyEntity") {
          //network request of some type?
          NSManagedObjectID *oid = [self newObjectIDForEntity:fRequest.entity referenceObject:@"SERVERS_UNIQUE_OBJ_ID"];
          //not shown: in-memory sorting, filtering, etc., see the class reference for a full scope of things you should be handling if the backend does not
          //not shown: caching
          return [NSArray arrayWithObject:oid];
         }
         else { return nil; }
      }
      else { return nil; }
    }
    else return nil;
} 

Note that you are responsible for handling all filters and sorts set on the fetch requests (if your backend does not).  This is a lot of work, and I am considering some way to abstract this work so that there is a “fall back” in-memory sort and filter implementation that can be re-used between NSIncrementalStore subclasses, but I have not yet thought through all the details.

Note that I am returning nil, but (oddly) not setting the error. Why is this? Because due to an undocumented bug in NSIncrementalStore (rdar://10732696), you cannot reference an error created inside an executeRequest:: call from outside the call. So if you do this:

- (id)executeRequest:(NSPersistentStoreRequest *)request withContext:(NSManagedObjectContext *)context error:(NSError  **)error {
  if (error) *error = [NSError errorWith...];
  return nil;
} //*error goes out of scope and is now dealloced up by the system, in spite of the fact that the caller in theory has a strong reference to it.

If you want your errors to survive the end of the method, you must make a manual call to objc_retain when you create them. Yes, really. File a duplicate bug with Apple.

Faulting

Allright, so now we’ve returned a set of fault objects to our application. But what happens when our application tries to access the value of a property? Our NSIncrementalStore class needs a new callback:

- (NSIncrementalStoreNode *)newValuesForObjectWithID:(NSManagedObjectID *)objectID withContext:(NSManagedObjectContext *)context error:(NSError *__autoreleasing *)error {
    //not shown: fetching myPropertyValue from network or cache
    return [[NSIncrementalStoreNode alloc] initWithObjectID:objectID withValues:[NSDictionary dictionaryWithObject:"myPropertyValue" forKey:@"myPropertyName"] version:1];
}

This method handles the fault fire of our managed objects (at least for property values). A similar callback exists that fires faults for relationship accesses.

A warning about threads

Generally speaking, when you use CoreData, you are interacting with local queries.  So calling managedObject.property is not terribly slow, and is something that can probably reasonably be done on the main thread.

However, when you introduce expensive remote queries into the mix, it’s not so simple.  Suddenly, accessing the property of an object can take a few seconds, or can potentially not be available if you’ve retired to your concrete-enforced nuclear bunker (or if the wind changes direction and you’re using AT&T).

Since you probably don’t want to do network requests on the main thread, you are going to be thinking about how to move CoreData code into the background.  CoreData threading is a scary and mystical topic, and I’ve talked to plenty of smart people who don’t really understand how it works.

The “cliff notes” answer is that everything is threadsafe except NSManagedObjectContext (e.g. the thing you use to run queries) and NSMangedObject (e.g. your entities themselves).  If you are doing cool things with threads, you need to make sure that you have a separate context for each thread and that you do not pass NSManagedObjects between threads.  You can serialize an NSManagedObject to its ID and then get a new NSManagedObject from the new context using the ID.

But a better pattern for most simple applications might be to do all CoreData access on its own thread and use callbacks to retrieve the results of queries.  I am considering adding some support for this in my CoreDataHelp project.  For example:

[CoreDataHelp executeFetchRequest: ... withCallback:^(NSArray* results, NSError *e) { ... }]  //an async method that calls you back with the results, which are automatically installed in your thread's MOC

[CoreDataHelp bgFetch:(id) managedObject key:(NSString*) key onCompletion:^(id value)]; //calls you back with the result of the property fetch

I’m still playing around with the syntax.  I like the first method quite a lot, but I have mixed feelings about whether wrapping a property fetch is a good idea in practice (or if I should try to rely on pre-caching the property values and only faulting to cache).  It may take me a few applications to settle on a pattern.

In conclusion

NSIncrementalStore is one of the best-kept secrets in iOS 5.  It has the potential to really change how we interact with remote data on iOS.  Today, integrating with web services on iOS is complicated, and doing anything nontrivial requires importing three different (conflicting) JSON parsers, dealing with SOAP fail, importing laughably bad libraries like ASIHTTPRequest that everybody seems to use for everything, doing low-level NSURLConnection hacking that is (at best) very hard to do correctly, and much more.  You need about fifteen tabs of documentation open on a second monitor just to get anything done, let alone how badly your application code gets polluted with backend implementation details.  Not to mention that one service you’re using that has taken it upon itself to produce its own poorly-written ObjC bindings that somehow has made the exact opposite design choices that you would have made for your application, and so you have to fork and hack it pretty aggressively to make it work, changes which will never make it upstream.

Now consider the alternative:  we start treating the NSIncrementalStore subclass as the way to wire up remote services to Cocoa.  (Either the remote service or the community starts wrapping up GitHub, Flickr, et. al. into NSIncrementalStores as the recommended way to talk to remove services.) Now you don’t have to think about whether they’re really using REST or only sort-of using REST, or whether their JSON really parses or which set of requests you need to make to get the data you want.  All you have to do is understand the object graph they expose (e.g. read a couple of class references), write a couple of NSFetchRequest calls, and you’re done.  The application interface is the same for every service.  This radically simplifies the complexity of writing API-consuming applications.

All the players in the iOS backend space (Parse, Kinvey, and Stackmob) are betting the farm that you would rather talk with Cocoa objects than worry about REST.  And they are right.  But with NSIncrementalStore, you can graft object support quite beautifully onto arbitrary, existing APIs.  And not only that, but you get the power of decades of improvements to CoreData out of the box (like faulting, uniquing, a powerful query language, batched queries, undo support, and much, much more).

But I think that Parse et. al. have much to worry about.  NSIncrementalStore is a complex beast, not for lesser developers, but it will gain a lot of mindshare on account of the fact that you can shoehorn existing APIs into it, something that will never be possible with cloud providers.  These two forces will cancel each other out.  Meanwhile, developers will be concerned about the long-term viability of businesses like Parse, or vendor lock-in, and if they have experience maintaining NSIncrementalStore code for GitHub et. al. they may in many cases consider NSIncrementalStore+REST a better candidate for new APIs than a proprietary service, in spite of its additional complexity.  Plus you get decades of cool CoreData tech for free.  (One area in which Parse et. al. can still win is through thread management, which is bad with CoreData as it was originally built for fast local queries, but can be fixed through appropriate and lightweight helper classes).

A call for NSIncrementalStore subclasses

So the next time you are reading Yet Another API Manual for consumption in your iOS / Mac application, consider whether the right response is to push an NSIncrementalStore subclass up to GitHub, even if it only supports a small subset of the remote API.  API bindings are an area that can really benefit from collaborative, open-source development, even if you are working on an otherwise proprietary Cocoa project.  I know that as I consume APIs going forward, I will be spawning lots of OSS NSIncrementalStore projects as a much better way to encapsulate interacting with remote data.

---

Want me to build your app / consult for your company / speak at your event? Good news! I'm an iOS developer for hire.

Like this post? Contribute to the coffee fund so I can write more like it.