Comments on: The Perils of Text-Only Search

By: Dave Kellogg

Dave Kellogg — Wed, 02 Sep 2009 23:45:57 +0000

Hi Seth,Thanks for reading. Indeed on your first point, I agree, the pages are in one sense new, because they are probably updated, but then again they are in another sense, old. For example, RSS seems to do a great job (and I'm not sure how) at only showing one post — the most recent — in my feed. And knowing that it's the same post, just modified. Google seems to lose track of when a page is the same page, just updated, vs. a new page. And I've embarrassed myself about 10 times at Mark Logic sending mails to the whole company saying "holy cow, breaking news, did you know that" [insert something that happened 2 years ago.]As for search syntax in Google, I'm not sure there's an easy answer for my trival problem with two adjacent words spanning paragraph.And, as I'm sure you know, my real point is that that is a *trivial* example of a much broader point on XML-awareness in content.I checked here for a solution on Google and, on a quick glance, didn't see one.http://www.google.com/support/websearch/bin/answer.py?answer=136861

By: Seth Grimes

Seth Grimes — Wed, 02 Sep 2009 15:08:40 +0000

Dave, of course –1) I'm guessing that the old pages for which you receive alerts *are* new: they're new to Google's index. I'd say that returning alerts for newly indexed content is valid although allowing you to specify you want only newly published content would be nice.2) I guess Google is doing greedy pattern matching, casting a wide net. I bet they have some level of regex support however. Have you tried being more precise? Seth

By: Dave Kellogg

Dave Kellogg — Wed, 02 Sep 2009 14:58:15 +0000

Yes and No.The server can be configured to ignore “superfluous tags” for phrases. In fact, by default we ignore many common HTML ones like the bold tag.So for a document with:helloworldYou can search for “hello world” and MarkLogic would find the document. We call this feature phrase-through.However, the behavior of the server is such that these tags act as word boundaries. Hence, tags through a word actually break the word into two:superbadSearch for “superbad” would miss. Search for “super bad” would hit.I’d call this capability word-through, though we don't currently support it.

By: Anonymous

Anonymous — Wed, 02 Sep 2009 12:12:28 +0000

Interesting! I think you covered most cases, except one, where you want the system to ignore the structural tag. Can you set Mark Logic to ignore presentation-level tags inside of words like em? Will Mark Logic find a hit for a search on "superbad" if the source text breaks up the word as follows:[b]super[/b]bad (superbad)? (I'm using square instead of angle brackets.)Can you set Mark Logic to ignore such "superfluous" tags in fulltext search?