The new Nook center keeps me away, I can tell you this much. I dislike it. More than that, it intimidates me. Yes, it intimidates me! I approach it hesitantly, suspiciously. Just looking at it makes me uneasy, and I actually avoid walking through it to get to the other side of the store. I’ve been trying to figure out why.

Three years ago a product of mine was undergoing a design review at Apple’s Worldwide Developers Conference. The review had been going smoothly until the engineer clicked on the toolbar icon for the application’s “Lexicon”. A new window appeared with a distinct interface and its own perspective on the underlying data. The UX engineer cringed. “You should consider moving this interface into the main window.” Qua? Here was a radically different way of looking at the data. It deserved its own window. “This window hides the main interface. You can’t tell where it’s gone, and it feels like you’re in a different application. I imagine my father doing this and being completely confused.” Well perhaps your father isn’t in the target market, I thought to myself at the time.

Of course the design engineer was right. By opening up such a different interface in an otherwise familiar application I was forcing the user to re-commit to the design. I had imposed upon him a world within a world, a new usability space that required re-orientation and exploration anew. The adjustment wasn’t an impossible one, or even necessarily a difficult one, but it did require a decision and commitment from the user which in turn imposed an additional cognitive burden. I imagine the gut response is unpleasant. “Great, now I have to figure this out too.” Or worse, the sudden, however brief, incomprehensibility of the space makes the user anxious, maybe even intimidates him.

The Nook center suffers from the same problem as my own Lexicon. Prior to its construction the Nook display occupied an open space near the entrance and cash registers. An impressive marketing presentation formed the backdrop to a couple of tables on which the various Nook models were laid out. Barnes & Noble had clearly taken a page out of the Apple Store book. Customers could come up, play with a Nook, ask the employees questions, and then browse on their way. But the presentation didn’t impose. Distinct just enough to attract your attention, it was still part and parcel of the store. It fit.

The new Nook center is a world unto its own. Like the Lexicon in its own window, the Nook center is enclosed in a separate space and isolated from the rest of the store. There are entrances at the four cardinal points, but walls otherwise close it off. And like the Lexicon with its unique interface, the Nook center exhibits a design motif that distinguishes it from everything else at the bookstore. Whereas the bookfloor is painted in earthy browns and greens, the Nook center is modern and metallic brimming with whites, grays and chromes. Whereas a walk through the store is pleasantly snug with packed bookshelves lining your path, the Nook center is oddly spacious. Finally the Nook center is bright. Very bright. Multiple LED screens beam marketing campaigns at you and the white interior reflects the fluorescent lighting in every direction. The rest of the store is dimmer in comparison and analogue. Posters of classic works line the walls and framed placards announce promotions. Unlike the other departments in the store, the cafe, the children’s area and the music section, the Nook center is utterly distinct.

Walking into the Nook center is like walking into another world. It’s as if you are leaving Barnes & Noble and entering another store. This requires a decision and commitment on the part of the customer. Am I finished browsing the aisles? Am I ready to leave? Am I ready to step into this store within a store and envelope myself in Nook, ready to reorient myself to its material and its surroundings? No doubt all of this is irrational and ridiculous. You’re just taking two steps into the Nook center and can leave it anytime you want. But it doesn’t feel like this. The Nook center is discontinuous with the rest of the store. It lies beyond an architectural and aesthetic threshold, and facing the decision to cross that threshold elicits a powerful emotional response.

Continuity or unity of design is a well established principle in interface engineering. The Nook center violates this principle with consequences. There is a cognitive burden associated with every decision to commit to a new environment, be it an architectural space in a bookstore or a user interface in an application. Usability patterns that require too much commitment from users will turn them off; the cognitive burden is too high and may not be worth it given their needs and their patience. The Nook center intimidates me because it is too different, because it requires too much of me to step into it.

It remains to be seen whether Barnes & Noble customers will acclimate to the new Nook center. Hesitation may eventually give way to curiosity and desire, patience will make due, and the center will be bustling with daily activity. The Nook is a fine product and deserves as much. In many cases, however, the designer may not have the luxury of time which acclimation requires. Users are fickle creatures, always on the lookout for better products at better deals. Stores and software alike may lose their customers if portions of the layout or interface are too distinct from the otherwise familiar context which surrounds them.

My immediate emotional response to the new Nook center taught me something that years of working with my own product couldn’t. Design must revolutionize, yes, but also integrate. For our part designers must think different and relate. The Nook center burdens its users with the unwanted commitment of the unfamiliar. We would do well to avoid the same mistake.

Initially I didn’t plan on open sourcing Journler until I was further along with the 2.6 update, but I finally decided I’d waited long enough and that it was time to publish the code even as I’m in the process of making significant changes to it. I’ve released not only the Journler code itself but also the three Sprouted frameworks used by Journler: SproutedUtilities, SproutedInterface and SproutedAVI. The frameworks are collections of shared code that I am deprecating, decomposing and working back into the main Journler project. The code is all live, in the sense that it is the code actually used in Journler and reflects the latest changes to the project.

Just this afternoon I also published the first bit of code out of Per Se. It is a custom NSTextContainer subclass with a generic algorithm for supporting complex text container geometry, enabling text wrapping around embedded rectangular content in any configuration along the x and y axis. It was one of the first classes I created for Per Se to see if building this kind of journaling application would even be possible in the first place. It is core code and forms one of three or four text system subclasses that implement Per Se’s custom page layout feature. Hot stuff.

All this code along with previously released code is being hosted in public repositories at github. You’ll find a complete listing at github.com/phildow.

Introduction

Matt develops three arguments to support his conclusion. The first concerns Frames of Interaction, the second Separation of Concern and the third System Integration, or native UI. In reality there is a single underlying premise here which Matt seems at times to recognize but does not genuinely pursue, namely, the premise that web apps must run in privilege restricted web browsers. I’d like to show that this premise is a false one and that once we recognize that web apps can run in dedicated environments, the distinction collapses and there is no real reason to favor a native platform over a web-based one.

I would like to emphasize, however, that my own conclusion speaks to a possibility. There will be no real reason. Dedicated environments for web applications with access to native hardware such as video cameras and audio inputs are to my knowledge not yet available for the iOS mobile platform. The point is they could be — the technology already exists on other platforms, including the Mac OS — and that we can instead read Matt’s conclusion not as a judgment against web based apps but as a call to action for that community of developers.

The Argument

Before Matt begins his critique he distinguishes between four different kind of apps. It is here that he first recognizes that a web app can in fact run within a web view inside a native shell. But Matt then ignores this possibility in the rest of his critique, which, as a number of his commenters have already pointed out, leads him to argue against a fictitious, exaggerated opponent. Matt’s own choice of imagery is a prime example of what’s called the straw man fallacy.

An Actual Straw Man

Straw Man on iPhone

Straw Man in UIWebView -- Looks awfully similar, doesn't it?

In his first Frames of Interaction argument Matt describes the increasing distance between a user’s actions and the interface supporting them as device layers are interposed between the user and the activity. We go from a calculator to a sleek calculator on a phone to a very ugly calculator in a distracting web browser on a phone. At each step the user’s focus is disrupted by the additional capabilities of the device. Web browsers confuse the app at hand by adding their own, unrelated “chrome”.

Of course this is based on the assumption that a web app must run inside a web browser, which is not at all the case. The iOS already provides the excellent UIWebView to host web based content in any manner the iOS developer wishes. All we need is a wrapper that displays the view in full screen and points it to a url. Doing so completely dissolves the browser layer and provides an environment similar to the native app. As I’ve mentioned Matt made note of this possibility but then he immediately ignores it.

The Separation of Concerns argument is an extension of the Frames theme. We generally have separate tools for different tasks: web browsing, emailing, document creation, etc. Not so with web apps, argues Matt. Web apps run in a single are of concern — the browser — and badly fracture the user’s focus by requiring her to switch back and forth between tabs and windows in a single application. This adds yet another level of interaction.  “We’re all guilty,” writes Matt, “of ignoring the fact that the browser is not the same as the device itself. It’s another level in the hierarchy, and you can’t pretend otherwise.”

Again, however, Matt is himself guilty of ignoring his own remark that web apps do not require browsers. As soon as the web application is hosted in its own environment, all of these issues disappear. There are no tabs, no windows, no additional hierarchy and no fractured attention. There is just the web app, as there is just the native app.

Matt’s final argument is the strongest of the three. Web applications, he says, must give up the native capabilities of the platform such as access to hardware including GPS and audio/video input. They must also give up integration with the rest of the platform. Web view wrappers can only go so far here, although Matt does finally recognizes the possibility of web-platform integration. He concludes that developers will choose to go all native because, basically, it’s harder to do hybrid.

Of course, it’s also harder to go the other direction and do web-integration with a native app. The iOS system frameworks don’t support Google docs or Google calendar or Facebook, Flickr, Twitter, Netflix, Hulu, etc. As users shift more and more of their digital existence to the web perhaps these will take priority over Apple’s own products, much, and already, no doubt, to Apple’s chagrin. But the present point stands. What about a web app’s access to Apple’s own platform?

Finally, Matt ties in UI considerations as well. Users have an emotional attachment to their products — particularly it seems, Mac users — and they expect their applications to be well designed. Web apps just don’t have the look and feel of native apps. Cross-platform is a pariah in the Mac community.

There is, however, no intrinsic failure here. If a web app does not feel like it belongs on the iPhone, the fault lies with the designers, not with the platform. If web-based frameworks, be they css/javascript, php, ruby or even perl, are not yet available to replicate the look and feel of a native iPhone app, this is only because the work hasn’t yet been done.

A Call To Action

Which leads me to the matter at hand. Now I may have been too hard on Mr. Gemmell here. Matt is critiquing the normal state of affairs. Many web applications don’t have dedicated wrappers, and even when they do their access to hardware and platform features is difficult and restricted. So instead let’s read Matt’s critique as a call to action. What can be changed to make web apps the equal of their native counterparts?

Assuming that most web apps are in fact accessed through browsers, the first step would be to support a protocol that allows a web app to run in its own dedicated process. The web app would request this privilege from the user and the browser would then launch a separate process, a fully separate application, to host the web app. This process would be nothing more than a minimal browser, dropping the excess chrome, nav, address bar, bookmarks, etc and without escalated privileges. The Chrome web browser on the Mac already spawns separate “Google Chrome Renderer” processes for every open tab. Why not include the option to run a dedicated app? The user could then save this quasi-bookmark to his desktop.

We have now removed the middling fame of interaction, but what about access to the native features of the platform? Apple has already established a precedent for us on the Mac OS. The WebKit framework supports two-way JavaScript and Objective-C communication using WebViews and WebScriptObjects. Objective-C can execute JavaScript code on a WebView’s currently loaded page and the JavaScript in that page can with an awesome ease call Objective-C code. There are also 3rd party frameworks that accomplish similar goals, such as JSCocoa, PhoneGap and QuickConnect.

With fully supported two-way communication between the web hosted environment and the native environment it should be possible to access the platform specific features of the iOS device. Web apps that target the iPhone environment can gather GPS data, activate the camera and record audio. If a comprehensive framework is lacking, only the effort is required.

Naturally, security concerns arise when such capabilities are introduced to web based applications which are not vetted by Apple’s own procedures. But again Apple has introduced a layer of security which could be adapted to these web applications: sandboxing. What is needed is a way for either web app developers themselves to securely sign their content for privilege escalation or for users to grant varying degrees of permissiveness to the sites they visit. At the least, a layer of security could be built into the platform’s web frameworks themselves.

Conclusion

If web-based applications are not yet the equal of their native counterparts this is only due to limitations which are not essential to the platform. When Mr. Gemmell focuses his criticism on applications hosted inside the web browser itself he ignores the greater possibility that web applications can, and eventually will, operate in their own frames of interaction with full access to the underlying hardware and the coveted look and feel of the iOS platform.

In fact, I believe this development is not far off. It is worth considering why Matt originally wrote his post. The motivation was the Financial Times’ move to a web platform, thus replacing its native iPad app. Might a talented individual who has dedicated so much of his time to the Macintosh platform find this threatening, the possibility that in the not too distant future, his — our — beloved NS, UI and Cocoa frameworks may be replaced by equally capable, more widely accessible web based frameworks?

I recently experienced a shock similar to this one attending a local Cocoaheads meeting. I’ve been developing desktop Macintosh applications professionally for over seven years now and have used and loved Apple products since my first Apple IIe computer. I was astonished, you can imagine, to discover that I was the only desktop developer at the meeting. Every other individual developed exclusively for iOS.

I realized then that the desktop developer is a dying breed. The tween and teen years of the 21st century mark the rise and eventually dominance of the mobile platform. Changes in the Mac OS already reflect this. But if such an evolution can come about in only seven short years, how long can we expect propriety mobile platforms to survive with the rise of web-based content, platforms and social circles?

And so I end this post with an unsettling if hyperbolic conclusion: that in another seven years Objective-C and the Mac and iOS frameworks which are so wonderfully wedded to it will all be dead, replaced by powerful, equally capable, just as beautiful and universally accessible web-based frameworks, perhaps unlike any of the frameworks we are familiar with now. And when that day arrives it will not be a question of preferring native apps over web-based counterparts.

Practically speaking, a page break allows a user to begin writing on a second page without filling up the first and without pressing the return key a dozen times to get to it. Its flexibility allows the user to then return to the first page and write without shifting the text on the next page, which otherwise occurs when a series of newlines (return keys) are used to simulate a page break.

Page breaking has been a widely requested feature throughout Per Se’s beta testing. Users would like to leave the left side of the journal for images or notes and write exclusively on the right side, or they would like to begin another topic on the following page, leaving the remainder of the current page empty.

It is a reasonable request. It is more than reasonable; it is expected, and I had put it on the to-do list for a future update without realizing how significant that expectation was. I also believed, wrongly as it turns out, that page breaking would be a difficult addition to make. I decided to postpone any research, not wanting to modify Per Se’s code in a large way so close to the release date and lacking the time to investigate the feature properly.

Interfaces
I only realized the necessity of page breaking in Per Se after reading the chapter on metaphors and idioms in About Face, The Essentials of Interaction Design. I had been reading about storage and retrieval systems when I was referred to this chapter, in which the author describes three interface paradigms: metaphoric, implementation-centric and idiomatic, criticizing the first two in favor of the third.

Metaphoric interfaces mimic real world objects, telephones, for example, or folders, filing cabinets, notebooks, planners and so on. They have become hugely popular with the rise of the iPhone, iPad and touch technology, as the tactile interaction establishes a similarity between software systems and physical objects. If the software looks and attempts to behave like its real world counterpart, it is metaphoric.

Idiomatic interfaces introduce easy to learn relationships between interactive elements and their functionality. Unlike metaphoric interfaces there is no pre-established relationship between the design and the system’s behavior. Idiomatic interfaces don’t mimic real world objects, and the relationships between appearance and function are arbitrary. A hyperlink is a good example. There is nothing inherent about underlined blue text which suggests clicking it will take you to another web page. But once you have learned this behavior, it is easy to recognize and apply yourself.

Implementation-centric interfaces base the design on the internal workings of the system. The layout is ordered such that it mirrors the organizational structure of the system it is presenting.

Journals & Journaling Software
Most journaling software is implementation-centric. Writings are organized by date and by list, maybe by folder as well. This is exactly how the software’s object model is organized. An entry object is associated with a date, and many entries may appear on a single date. The user needs a way of accessing those dates and selecting one of the entries which occurs on it. If there are folders, there is also a container object which refers to any number of entries, one of which must also be selected from a list for viewing and editing. The on-screen presentation explicitly duplicates the software’s data organization pattern, so that you see a calendar, folders, and a list of entries.

But is this how we actually keep a journal? Hardly. I certainly don’t look at a calendar on the inside front cover which refers me to the correct page for a given date. And I rarely even use the date when I’m looking for an entry I previously wrote, partly because the date may have been a matter of complete indifference to the topic, partly because I don’t index or associate in my mind entries with a specific date.

If I am creating a new entry, mostly I just open up my journal to the next blank page and start writing. I note the date only because it is the most immediate means of organizing the journal, not because it is the most significant. When I’m looking for a previously created entry, often it is by content or by topic: what I wrote or that about which I wrote. Folders help here as well as tags, which are just folders that collect their content by topic automatically.

But knowing what I’m looking for — if I am so fortunate to even have this — and recognizing that I’ve found it are two different things. If I have a hundred or more entries on a given topic, knowing that I’m searching for an entry about this topic hasn’t yet found it for me. I still have dozens of potentially lengthy writings to search through.

When I’m perusing my written journal, seeking out a specific entry, often, and somewhat surprisingly, it is not the content which provides the most important cue to recognizing the entry I’m looking for, but its appearance, how I placed the text on the page, whether I left a lot of empty space or wrote in a box, used arrows and diagrams, added images and so on. It is the form rather than the material by which I recognize an entry, even though it is the material I seek. And this has nothing to do with calendars, lists and folders.

Per Se & The Importance of Page Breaking
Per Se obviously establishes a metaphoric interface. I didn’t explicitly set out to create this kind of interface when I first envisioned the application. Really I was just jealous of the gorgeous interfaces being developed for portable, tactile platforms and wondered why they couldn’t also be implemented on the Mac. I was also reacting against the currently dominant paradigm in Mac journaling software and was hoping to create a unique product.

The author of About Face is strongly critical of metaphoric interfaces. He enumerates a number of limitations, many of which I must take into consideration as I continue to develope Per Se, but the one which immediately caught my attention concerns the expectations that metaphors engender:

"After the user depends on the metaphor for recognition, he expects consistency of behavior with the real-world object to which the metaphor refers."

Per Se mimics a physical journal. With a physical journal it is possible to use both sides of the open page without filling up one side first. Per Se in its current form (RC2) permits this but only incompletely. For example, Per Se supports text wrapping around images. And while it is possible to place images, notes, flourishes, files and any other kind of attachment on the right side of the journal without filling up the left, it is not possible to actually write there. Text layout is “linear and continuous”, as I’ve told a dozen beta testers. You must fill up the left side of the page first.

Per Se establishes and immediately breaks the metaphor of the physical journal.

I don’t think it’s possible to over-emphasize the importance of meeting user expectations when designing software. If an interface begets expectations and they are not met, it is frustrating. Very frustrating. So much so that users will stop using your software if it doesn’t work the way they expect it to. There are some expectations you cannot account for, but when the design you have created establishes expectations which it does not meet, you are wholly responsible for the failure. And by you, I mean me.

Implementing Page Breaking
After reading that passage in About Face I decided to take another look at page breaking. I knew Text Edit permitted it, so I checked the sample Text Edit source code included with every copy of Apple’s developer tools. The menu sends the message insertContainerBreak: to the first responder. I couldn’t find it in Text Edit’s source, but a second quick lookup showed that it was an NSResponder method explicitly implemented by NSTextView to insert page break control characters.

Meaning page breaking is built into the system text objects. Meaning it requires exactly zero lines of code to get this working in Per Se. Sure enough, after hooking up my own menu items to that first responder message, Per Se was supporting page breaks.

The implementation wasn’t quite right though. I had added the page break command to the Insert menu, which is where users find commands for links, checkboxes, tables, attachments and so on. In order to page break, you had to select the menu item, which is clearly not how users move to the other side of their physical journal. They just pick up the pencil and set it down on the opposite page.

The equivalent is clicking on the other page with the mouse. Responding to one click might be a bit unexpected and jarring, the more so if it isn’t obvious how to get rid of the page break, not because pressing the delete key or selecting Undo is difficult but because a single mouse click in empty space doesn’t generally have that kind of effect.

Instead, I opted for two clicks. With less than five lines of code, I am able to override NSTextView’s mouseDown: method, check for the presence of glyphs, or laid out text, on the page, and if there are none insert a page break. From the user’s perspective, all she has to do is double click on the empty right side of the journal and begin typing there. She can continue to place images and attachments however she wishes, and she can return to the left side and write without altering the position of the text on the right side.

All in all it works beautifully and as expected, and it required hardly a modicum of effort on my part.

It isn’t just a superfluous metaphor
Supporting page breaks is important for two reasons. While there may be enduring problems with metaphoric interfaces, software which nevertheless pursues this paradigm must pursue it as completely as possible or risk negating user expectations and losing customers. If Per Se is to be a successful program, it must work the way users automatically expect it to.

Completing the metaphor isn’t just a matter of internal consistency. As the author notes in About Face, metaphors are visual. When they work, they work because humans rely largely on visual cues when navigating their environment, and especially when recognizing objects in that environment. By ensuring that Per Se works the way users expect it to I also enable a more robust, natural way of recognizing data: by its appearance, rather than by its content or its temporal or hierarchical location on a date or in a folder. Which is quite a feat considering that Per Se is dealing mostly with text.

Custom, digital page layout that includes page breaking is a significant step towards completing the metaphor of the physical journal. It empowers the user in natural but unexpected ways. And once you’ve tried it, you just can’t help notice how cool it is.

[[ Originally posted at http://getsprouted.com , but it feels more appropriate here. ]]

This class has been factored directly out of Journler and brought up to date with the help of modern Objective-C 2.0 language syntax, Mac OS concurrency APIs and blocks programming. A download link is available at the bottom of this article, or you may go to the Source Code page for a link.

Summary

SPSearchStore is an Objective-C wrapper for the SearchKit API. SearchKit offers document indexing and “google-like” querying to Core Foundation apps. SearchKit is a stupid powerful Core Services API that supports threaded access, document to term and term to document mapping, text summarizing, and similarity, boolean, phrasal and wildcard prefix/suffix queries.

SPSearchStore makes most of the API accessible to Cocoa applications by way of a simple public interface. The class supports direct access to the complete two-way to-many documents / terms graph contained in a SearchKit index, establishing the foundation for more complex analysis based on the semantic relationships among an arbitrary collection of documents.

Like SearchKit, SPSearchStore is thread safe. You may use multiple threads to read from and write to the search store. In fact the class will itself use threads to import documents and query the index if you enable its concurrency option. SPSearchStore employs locks to manage access to the underlying data and Grand Central Dispatch by way of operation queue blocks to thread processing.

Workflow

Employing SPSearchStore is a two step process, with an optional third step to take advantage of the document / term graph. Each step can be accomplished with a minimal amount of code, as I’d like to show you here. Generally, you will 1. Establish a store, 2. Perform searches and 3. Perform document / term analysis.

1. Establishing an index

In the first step you create a store and add content to it. SearchKit supports both in-memory and disk based indices. The APIs manage persistence for the latter option, but you can also persist in-memory stores with just a little extra effort. For either store type you may specify a wide range of text analysis options including minimum term length, stop words, synonyms, starting term characters, and so on.

You can import two kinds of content into the store: file based content or free standing text that you supply. For file based content you may optionally have SearchKit use Spotlight importers for text extraction. In both cases SearchKit uses URIs to identify the content, and it is primarily with URIs that you’ll be interfacing with SPSearchStore.

A. Set up default text analysis options prior to store creation:

[SPSearchStore setDefaultTextAnalysisOption:[NSNumber numberWithInteger:2]
forKey:(NSString *)kSKMinTermLength];

B. Create a memory or disk based store with a single call:

searchStore = [[SPSearchStore alloc] initStoreWithMemory:nil 
type:kSKIndexInvertedVector];

C. You can then set store behavior:

searchStore.usesSpotlightImporters = YES;
searchStore.usesConcurrentIndexing = YES;

D. And add content to the store:

[searchStore addDocument:(NSURL*)obj typeHint:nil];

2. Performing a search

SearchKit uses a two stage pattern to perform searching, and that process is mirrored by SPSearchStore. The first stage initializes the search from a query / options combination and starts the search running on a separate thread. The second stage consists of potentially multiple calls to the index asking for chunks of the search results.

SearchKit is extremely flexible and quite powerful in its handling of queries. It supports similarity, phrasal, wildcard and boolean searching. Documentation describes the syntax as “Google-like”. Often, however, it will not be enough to simply pass a user generated string directly to SearchKit. Cocoa developers accustomed to the simplicity of predicate syntax such as contains[cd] will wonder why SearchKit doesn’t provide this kind of searching directly.

While diacrtical and case sensitive issues are automatically handled, SearchKit otherwise looks for exact matches unless you surround your strings with the wildcard character. For example, predicate syntax such as “contains[cd] tune” must look like “*tune*” to SearchKit. If your user expects this kind of functionality but doesn’t know or shouldn’t have to correctly format the string, it will be up to you to make the necessary adjustments.

Refer to the developer documentation for more information on SearchKit query syntax.

A. Initialize a store search from a query string and query options:

NSString *searchString = @"foo* && *bar";
[searchStore prepareSearch:searchString options:kSKSearchOptionDefault];

B. Fetch the search results either at one time or with multiple calls:

NSArray * results = nil;
NSArray * ranks = nil;
[searchStore fetchResults:&results ranksArray:&ranks untilFinished:YES];

C. Normalize the relevancy results:

NSArray * normalizedRanks = [searchStore normalizedRankingsArray:ranks];

3. Performing document / term analysis

SearchKit offers complete access to the two-way term / document associations in an inverted vector index. The SearchKit documentation only mentions vector indices in the context of similarity searching, but more generally, vector indices map documents to terms. With a vector index you can acquire a complete list of the unique terms contained within a document.

Normally, searching employs an inverted index approach which maps terms to documents. You specify a term and the search returns the documents in which the term appears. A combined inverted vector index maps both the term to document and document to term associations, allowing you to go back and forth among document / term relationships.

With an inverted vector index you can acquire all the terms in the index, all the documents in the index, all the documents associated with a term, all the terms in a document, and the number of times a term appears in the index or in a document.

Imagine: the user queries for a term and you return a list of documents. The user selects a document and you display a list of the terms it contains. The user selects a term, and again you show the documents which match it, and so on. This is the principle behind Journler’s lexical capabilities and is, in a word, awesome. SearchKit rocks, and SPSearchStore secures straightforward access to these capabilities.

A. Get all the terms or documents in the search index:

NSArray *allDocs = [searchStore allDocuments];
NSArray *terms = [searchStore allTerms];

B. Get all the unique terms contained in a specific document:

NSURL *docURI = ...;
NSArray *docTerms = [searchStore termsForDocument:docURI];

C. Get all the documents which contain a specific term:

NSString *term = @"term";
NSArray *docs = [searchStore documentsForTerm:term];

Limitations

SPSearchStore provides access to most of SearchKit’s functionality, but there are a couple of noticeable limitations.

1. No support for document hierarchies: SearchKit supports the hierarchical indexing of document content, whether file based on free-standing text. SPSearchStore does not provide an interface to this mechanism.

2. No support for text summarization: SearchKit includes a set of APIs for summarizing documents. SPSearchStore does not support this functionality, choosing instead to focus on query and document/term capabilities. It should, however, be trivial to add a summarization category to NSString.

Concerns

In the past there have been bugs reported with SearchKit’s use of 3rd party Spotlight importers. There have also been more recent reports of memory issues with SearchKit. SPSearchStore could benefit from a comprehensive set of UnitTests for different document types and indexing conditions.

Perhaps most annoyingly, SearchKit only indices the textual content of files. It does not index any other metadata information nor does it index the file’s name. Users will probably expect searches to match file names, but this is something you must accomplish separately, probably using NSPredicate.

Conclusion

SearchKit is a powerful, flexible Core Foundation API that can be difficult for Cocoa programmers to wrap their heads around. SPSearchStore is an Objective-C class which captures much of the SearchKit functionality. With a minimal amount of coding Cocoa programmers now have access to this incredible API.

SPSearchStore is available free of charge under the terms of a BSD license. If the terms of this license do not meet your needs you may optionally purchase a non-attribution license instead. For more information, refer to the Licensing page.

Download SPSearchStore