From the blog of Eric Hellman

Triple Stores Aren’t

A Reflection on the Structure and Process of the Web of Data

Google Rolls out Semantic Search Capabilities

“Google has given its Web search engine an injection of semantic technology, as the search leader pushes into what many consider the future of search on the Internet.”

Part on an email message from Art.Pasquinelli@sun.com

The Internet Archive and the Sun Modular Datacenter Announcement – March 25
Sun and the Internet Archive rolled out a joint project at a March 25 event in Santa Clara. Sun has approximately 60 Sun Fire x4500s managing the Internet Archive’ content in a Sun Modular Datacenter. Dozens of articles were generated globally (see below). We will have a presentation and consulting on the architecture at the June 24-26 Sun PASIG in Malta (Early Bird registration will be open in the next 1-2 days at www.sun-pasig.org).

The video tour and Sun case study can be seen at; http://www.sun.com/featured-articles/2009-0325/feature/index.jsp

KEY QUOTES

“’Archive.org also houses the Wayback Machine, 1 million books, 100,000 movies and about 200,000 audio recordings,’ Kahle said. ‘It is a full-on library. This technology we see as another step toward a manageable system for dealing with enormous amounts of information safely.’” – eWEEK

“Each container packs in 60 of the company’s Sun Fire X4500 Open Storage Systems and is constantly monitored for potential threats. It’s actually a pretty elegant, modular solution to an archive that grows by nearly 100TBs every month.” –Gizmodo

“’For years, the folks at the archive got by building their own open-source computer systems,’ said Brewster Kahle, its founder — at one point the archive spun off a company to build systems — but they couldn’t move fast enough to keep up with advances in technology or the growth of the Web. So today Sun Microsystems will announce that the archive has migrated its digital library to a single Sun data center housed in a shipping container on Sun’s campus in Santa Clara.” – San Francisco Chronicle

“’It may be the single largest database in the world, and it’s all in a shipping container. I think of the shipping container as a single machine or expression made up of many smaller machines,’ said Brewster Kahle, digital librarian and co-founder of the Internet Archive, the nonprofit organization that runs the Wayback Machine site.” – Computerworld

“The Internet Archive is moving its data to one of Sun’s datacenter-in-a-box units, and not a minute too soon. At the rate that print is dying, we’ll need to step up the archival pace if we’re going to retain our cultural artifacts in the digital age.” – Ars Technica

AllegroGraph

We chose to give it special attention for reason which include, but are not limited to that it is:

• Commercially supported, and because Franz is very active in the Sematic Web community
  Last year being a fairly major sponsor of the Semantic Technology Conference

• Free to us at the level we currently have need for it
  Franz has even generously provided a significant amount of free technical support

• Actively maintained and updated by Franz

• Supports SPARQL

• Seemed to have some of the best benchmark performance results

• Has a Java-based API and is compatible with Jena and Sesame

• Goes beyond simple subject/predicate/object triple-based support
  Implements statementID’s and Named Graph entries

• Can bulk load from RDF/XML and N-Triples

• Supports direct generation of JSON from SPARQL queries

• Offers Free-text indexing

• Supports clustering and federation

• Franz is also very much into artificial intelligence and reasoning,
  although those are beyond the scope of our current interest.

However, in spite of all the nice features and attractions listed above, we had trouble with AllegroGraph when it came to managing various combinations of concurrent usage, including attempts to perform unregulated:

• Reads
• Writes
• Re-indexing

especially if those involved multiple simultaneous users.

Consequently, we embarked on a fairly serious attempt to analyze the performance capabilities of AllegroGraph, and this in turn led us to begin studying similar performance of other triplestore implementations, including: Oracle, Sesame, Mulgara.

Doing this analysis was a challenge though because there were a series of updates to the products which meant we had a sort of moving target to work with.  For instance, at some point Franz updated from the AllegroGraph 2 to AllegroGraph 3 series and in one of our important tests we observed a performance improvement of a factor of over two orders of magnitude reduction in query response time.

I must note that we also had other tasks and priorities which demanded our attention and distracted us from the investigation.  Thus we have not yet managed to either complete our testing, or actually migrate to the latest AllegroGraph (version 3.2 as of this writing).

In short we continue to have problems managing the activities of reading, writing and indexing and have gone to some complicated lengths to separate these activities.  In particular, we have dedicated an independent AllegroGraph server and instance to write operations and then have tasks to synchronize the writeable instance of AllegroGraph to a read-only instance on a daily basis.  The intent of this separation is to protect the read-only version from write and indexing operations which can impair its performance.

Further, because we still have concerns about triplestore (AllegroGraph) performance in our Production environment, we also synchronize particular triplestore data to a Solr instance, which we actually use as our dominant live/active Production query source.  This leave the read-only triplestore protected for more specialized SPARQL query usage.

We have been in frequent contact with Franz about our issues with AllegroGraph’s abilities to handle concurrent activity.  Their tech support staff have been great in trying to help us, but have also acknowledged some of AllegroGraph’s limitations in this area.  They have told us that they would be working on improving some of the problems we have observed, and in fairness, we have not been able to keep up on properly testing their latest releases.

That is a quick summary of our situation.  I’d be very interested to hear how others might be dealing with similar issues.

Thanks.

Chris Frymann
Digital Library Architect
UC San Diego Libraries

The above is a variant of a message originally sent to Ben Osteen

Is the Relational Database Doomed?

Discusses some of the potential of key/value databases as compared to RDBs.   The immediate answer to the inflammatory title is, of course, no.  However, it seems increasingly clear that one can find a lot of company in suggesting that there is significant value in and adoption of schema-less database approaches.

See also:

How FriendFeed uses MySQL to store schema-less data

There are many responses to the above post, so some reading is required, but it may be worth it.  I found it interesting though that there were no comments on triplestores.  I’m not quite ready to jump in on that though.  I’ll try and come back to it later and see what additional comments may have been made.

In any case, performance is still always an issue.  Many approaches are taken to dealing with response time, and some form of replication is frequently involved, whether it be copying data into parallel systems,  or storing it in alternate forms or formats that have different access versus update characteristics.  This inevitably leads to a problem of maintaining consistency between the various manifestations of the data.  The challenge of maintaining consistency across various forms of parallel systems is therefore a recurrent theme and one addressed in the following sources:

Eventually Consistent – Revisited

Discusses some of the problems managing reads and writes and keeping everything consistent.

Sesame 3.0 Preview: An Open Source Framework for RDF Data

From a recent DevX.com article. Mentions the concept of “eventual consistency” in the “Transactions” section.

In our case, end-user results and the process of achieving consistency depends on the order in which one updates:

• Files
• Triplestores
• Solr Indexes

The following references and quote are from an email exchange with Benjamin O’Steen [bosteen@gmail.com].

Writing to serialized data (files), and later updating Solr indices using JMS/AMQP [RAbbitMQ] enables ” indexes ‘eventually converging’ to the truth within seconds after the event (truth being whatever the data held on disc says is true.)”

Changes to an RDF document can be queued as a Talis changeset and later committed.

Note: This post originally addressed the topic of R/W Contention in triplestores and Solr as an approach, posed by Declan Fleming.