Archive

coding

There are any number of reasons that you can attribute to Solr‘s status as the standard bearer of faceted full-text searching:  it’s free, fast, works shockingly well out of the box without any tweaking, has a simple and intuitive HTTP API (making it available in the programming language of your choice) and is, by far, the easiest “enterprise-level” application to get up and running.  None of its “competitors” (Sphinx, Xapian, Endeca, etc.), despite any individual advantages they might have, can claim all of these features, which goes a long way towards explaining Solr’s popularity.

The library world has definitely taken a shine to Solr:  from discovery interfaces like VuFind and Primo, to repositories like Fedora, to full-text aggregators like Summon, you can find Solr under the hood of most of the hot products and services available right now.  The fact that a library can install VuFind and have a slick, jaw-droppingly powerful OPAC-replacement that puts their legacy interface to shame in about an hour is almost completely the by-product of Solr’s amazing simplicity to get up and running.  It’s no wonder why so many libraries are adopting it (compare it to SOPAC, also built in PHP and about as old, but uses Sphinx for the full-text indexing and is hardly ever seen in the wild).

Without a doubt, Solr is pretty much a no-brainer if you are able to run Jetty (or Tomcat or JBoss or Glassfish or whatever):  with enough hardware, Solr can scale up to pretty much whatever your need might be.  The problem (at least the problem in my mind) is that Solr doesn’t scale down terribly well.  If you host your content from a cheap, shared web hosting provider or a VPS, for example, Solr is not available or not practical (it doesn’t live in small memory environments well).  The hosted Solr options are fairly expensive and while there are cheap, shared web hosting providers that do provide Java Application Servers, switching vendors to provide faceted search for your mid-size Drupal or Omeka site might not be entirely practical or desirable.

I find myself proof-of-concept-ing a lot of hacks to projects like VuFind, Blacklight, Kochief and whatnot and run these things off of my shared web server.  It’s older, underpowered and only has 1GB of RAM.  Since I’m not running any of these projects in production (just really making things available for others to see), it was really annoying to have Solr gobbling up 20% of the available RAM for these little pet projects.  What I wanted was something that acted more or less like Solr when you pointed an application that expected Solr to be there, but I wanted it to have a small footprint that could run (almost) anywhere and more or less disappear when it was idle.

So it was for this scenario that I wrote CheapSkate: a Solr emulator written in Ruby.  It uses Ferret, the Ruby port of Lucene, as the full-text indexing engine and Sinatra to supply the HTTP API.  Ferret is fast, scales quite well and responds to the same search syntax as Solr, so I knew it could handle the search aspect pretty easily.  Faceting (as can be expected) proved the harder part.  Originally, I was storing the values of fields in an RDBMS and using that to provide the facets.  Read performance was ok, although anything over 5,000 results would start to bog down – the real problem was the write performance, which was simply woeful.  Part of the issue was that this design was completely schemaless:  you could send anything to CheapSkate and facet on any field, regardless of size.  It also tried to maintain the type of the incoming field value:  dates were stored as dates, numbers stored as integers and so on.  Basically the lack of constraints made it wildly inefficient.

Eventually, I dropped the RDBMS component, and started playing around Ferret’s terms capabilities.  If you set a particular field to be untokenized, your field values appear exactly as you put them in.  This is perfect for faceting (since you don’t want stemming and whatnot on your query filters and your strings aren’t normalized or downcased or anything so they look right in the UI) and is basically the same thing Solr itself does.  Instead of a schema.xml, CheapSkate has a schema.yml, but it works essentially the same way:  you define your fields, what should be tokenized (that is, which fields allow full-text search) or not (i.e. facet fields) and what datatype the field should be.

CheapSkate doesn’t support all of the field types that Solr does, but it supports strings, numbers, dates and booleans.

One neat thing about Ferret is that you can pass a Ruby Proc to the search method as a search option.  This proc then has access to the search results as Ferret is finding them.  CheapSkate uses this find the terms in the untokenized fields for each search hit, throws them in a Hash and generates a hit count for each term.  This is a lot faster than getting all the document ids from the search, looping them and generating your term hash after the search is completed.  That said, this is still definitely the bottleneck for CheapSkate.  If the search result has more than 10-15,000 hits, performance begins to get pretty heavily impacted by grabbing the facets.  I’m not terribly concerned by this, data sets with search results in the 20,000+ range start to creep into the “you would be better off just using Solr” domain.  For my proofs-of-concepts, this has only really raised its head in VuFind when filtering on something like “Book” (with no search terms) for a 50,000 record collection.  What I mean to say is, this happens for fairly non-useful searches.

Overall, I’ve been pretty happy with how CheapSkate is working.  For regular searching it does pretty well (although, like I said, I’m not trying to run a production discovery system that pleases both librarians and users).  There’s a very poorly designed “more like this” handler that really needs an overhaul and there is no “did you mean” (spellcheck).  This hasn’t been a huge priority, because I don’t really like the spellcheck in Solr all that much, anyway.  That said, if somebody really wanted this and had an idea of how it would be implemented in Ferret, I’d be happy to add it.

Ideally, I’d like to see something like CheapSkate in PHP using Zend_Search_Lucene, since that would be accessible to virtually everybody, but that’s a project for somebody else.

In the meantime, if you want to see some examples of CheapSkate in action:

One important caveat to projects like VuFind and Blacklight:  CheapSkate doesn’t work with Solrmarc, which requires Solr to return responses in the javabin format (which may be possible to hack out something that looks enough like javabin to fool Solrmarc, I just haven’t figured it out).   My workaround has been to populate a local Solr index with Solrmarc and then just dump all of the documents out of Solr into CheapSkate.

A couple of months ago, I hacked up a really simple proof-of-concept Sinatra that took an LCCN, called the Library of Congress’ LCCN Permalink service’s MARCXML output for that particular LCCN and tried to model it into linked data. It was really basic: it only returned RDF/XML and had no persistence layer to it, so I ran it using Heroku’s free hosting plan.

It worked pretty well but as I applied more and more functionality to it (looking for matches in Musicbrainz, LinkedMDB, DBpedia, Freebase, etc. — especially inefficient SPARQL regex queries), I kept running into execution timeout errors on Heroku. These are the exact same sorts of problems that the Umlaut ran into years ago and the solution required complicated threading and, eventually, AJAX requests to offload some of the response time of waiting for synchronous web services requests to return (or timeout or fail).

One thing I began doing with LinkedLCCN was persisting the graph into a Platform store, so once stored, any subsequent request for a resource was quite speedy. The problem was the initial request, the one that gathered all of the data to fill out the graph before it stored. Quite often this would timeout or throw an error (which, given that this was still very much a work in progress, would result a 500 error) meaning the resource was never saved to the Platform meaning all of the following requests would have to go through the same process until one of them finally succeeded. Since the freebie access on Heroku lets you run one process at a time, these long running (and timing out) requests would cause a backlog which would throw more errors.

It was becoming the embodiment of the phrase I used during my Code4Lib presentation: “Amateur Hour on the Internet”.

What was obviously needed was some asynchronous mechanism for giving back part of the graph, indicating to the requester that this was a partial response, and firing off a background task to complete the rest of the processing. Because there was no HTML interface, AJAX wasn’t an option. Even if there was an HTML interface (as there is now), AJAX still wouldn’t have been an option, because this was a service intended for web agents following their nose, not human surfers, so even if the agent is satisfied with the HTML response (for instance, when it eventually gets RDFa), curl (and its ilk) don’t have javascript, so the background process would never even have a chance to be called, anyway.

This meant the only viable solution to this problem was going to be via multiple processes. This also meant that Heroku wasn’t an option anymore (at least, not without a price), so I was going to migrate to my personal web host. In Ruby web frameworks, asynchronous processing comes in one of two forms:

  1. Threads/forks/etc.
  2. Queue schedulers

Based on my experience with the Umlaut, I wanted to avoid #1, if at all possible, or at the very least, use an existing, packaged solution that could drop fairly painlessly into Sinatra. I found a port of Merb’s run_later, but I could only ever get it to run once. Any succeeding request never seemed to fire off the process.

The queue schedulers generally required their own set of baggage: namely their own running daemon and a RDBMS. Since almost all of these projects originally started out as Rails plugins, they expect the application to have ActiveRecord and an RDBMS to store their jobs in. I didn’t have either.

I had settled on using Delayed Job, since there was, again, a Sinatra port. It took quite a bit of hacking to get this to work right (mainly around marshaling/unmarshaling objects), and I never could get the job logging to work very well, but it was successfully queuing and executing jobs in the background.

It was hard to manage, though. I use Capistrano for deployment and it was very difficult to control the Delayed Job daemon so that it would stop and start with the regular webservice. Again, it worked, but it felt very fragile. The sort of thing I could see breaking and having to spend hours trying to figure out how to fix it.

Last night, while I was trying to pull together my thoughts and links and whatnot for this post, I ran across Spork, which is a Sinatra port of Spawn. A couple hours later, LinkedLCCN was refactored to use Spork instead, and that’s how it’s running now.

So, LinkedLCCN now works like this:
$ curl -v -H “accept: application/rdf+xml” http://lccn.lcsubjects.org/93707283#i

* About to connect() to lccn.lcsubjects.org port 80 (#0)
*   Trying 208.83.140.6... connected
* Connected to lccn.lcsubjects.org (208.83.140.6) port 80 (#0)
> GET /93707283#i HTTP/1.1
> User-Agent: curl/7.19.6 (i386-apple-darwin9.8.0) libcurl/7.19.6 zlib/1.2.3
> Host: lccn.lcsubjects.org
> accept: application/rdf+xml
>
< HTTP/1.1 206 Partial Content
< Date: Fri, 12 Mar 2010 20:19:50 GMT
< Server: Apache/2.2.12 (Ubuntu)
< X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.11
< Content-Length: 6472
< Status: 206
< Content-Type: application/rdf+xml
<
<rdf:RDF xmlns:dcterms="http://purl.org/dc/terms/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:foaf="http://xmlns.com/foaf/0.1/" xmlns:skos="http://www.w3.org/2004/02/skos/core#" xmlns:bibo="http://purl.org/ontology/bibo/" xmlns:owl="http://www.w3.org/2002/07/owl#" xmlns:umbel="http://umbel.org/umbel#" xmlns:rda="http://RDVocab.info/Elements/"><rdf:Description rdf:about="http://purl.org/NET/lccn/93707283#i"><rda:placeOfPublication><rdf:Description rdf:about="http://purl.org/NET/marccodes/countries/nyu#location"></rdf:Description></rda:placeOfPublication><rda:titleProper>The freewheelin' Bob Dylan</rda:titleProper><foaf:isPrimaryTopicOf><rdf:Description rdf:about="http://lccn.loc.gov/93707283"></rdf:Description></foaf:isPrimaryTopicOf><bibo:uri>http://hdl.loc.gov/loc.mbrsrs/lp0001.dyln</bibo:uri><bibo:lccn>93707283</bibo:lccn><dcterms:title>The freewheelin' Bob Dylan</dcterms:title><dcterms:creator><rdf:Description rdf:about="http://purl.org/NET/lccn/people/n50030190#i"><owl:sameAs><rdf:Description rdf:about="http://dbpedia.org/resource/Bob_Dylan"></rdf:Description></owl:sameAs><foaf:name>Dylan, Bob, 1941-</foaf:name><umbel:isAbout><rdf:Description rdf:about="http://viaf.org/viaf/46946176.rwo"><foaf:name>Dylan, Bob, pseud</foaf:name><foaf:name>Dylan, Bob, 1941-</foaf:name><foaf:page rdf:resource="http://dbpedia.org/page/Wikipedia:WikiProject_Bob_Dylan" /><foaf:page rdf:resource="http://www.worldcat.org/wcidentities/lccn-n50-030190" /><foaf:page rdf:resource="http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Bob_Dylan" /><skos:altLabel>Thomas, Robert Milkwood,</skos:altLabel><skos:altLabel>Dylan, B.</skos:altLabel><skos:altLabel>Thomas, Robert Milkwood</skos:altLabel><skos:altLabel>Landy, Bob,</skos:altLabel><skos:altLabel>Landy, Bob</skos:altLabel><skos:altLabel>Zimmermann, Robert Allen</skos:altLabel><skos:altLabel>Zimmerman, Roberto Allen</skos:altLabel><skos:altLabel>Zimmerman, Robert,</skos:altLabel><skos:altLabel>Porterhouse, Tedham,</skos:altLabel><skos:altLabel>Petrov, Sergei</skos:altLabel><skos:altLabel>Zimmerman, Robert Allen,</skos:altLabel><skos:altLabel>Blind Boy Grunt,</skos:altLabel><skos:altLabel>Gook, Roosevelt,</skos:altLabel><skos:altLabel>Dylan, Bob, 1941-</skos:altLabel><skos:altLabel>Zimmerman, Robert</skos:altLabel><skos:altLabel>Alias,</skos:altLabel><skos:altLabel>Zimmerman, Roberto Allen,</skos:altLabel><skos:altLabel>Dylan, Bob, pseud</skos:altLabel><skos:altLabel>Zimmerman, Robert Allen</skos:altLabel><skos:inScheme rdf:resource="http://viaf.org/viaf-scheme/#personalNames" /><skos:inScheme rdf:resource="http://viaf.org/viaf-scheme/#concept" /><skos:changeNote xml:lang="en">Modified by agency: OCoLC</skos:changeNote><skos:changeNote xml:lang="en">Transcribed by agency: OCoLC</skos:changeNote><skos:exactMatch rdf:resource="http://viaf.org/viaf/46946176.viaf" /><skos:exactMatch rdf:resource="http://libris.kb.se/auth/184248" /><skos:exactMatch rdf:resource="http://viaf.org/viaf/46946176.m21" /><skos:exactMatch rdf:resource="http://id.loc.gov/authorities/n50030190#concept" /><skos:exactMatch rdf:resource="http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Bob_Dylan" /><skos:exactMatch rdf:resource="http://viaf.org/viaf/46946176.unimarc" /><skos:exactMatch rdf:resource="http://viaf.org/processed/BNF%7C13893566" /><skos:exactMatch rdf:resource="http://viaf.org/processed/NKC%7Cjn20000700458" /><skos:exactMatch rdf:resource="http://d-nb.info/gnd/118528408" /><skos:exactMatch rdf:resource="http://catalogo.bne.es/uhtbin/authoritybrowse.cgi?action=display&amp;authority_id=XX821701" /><skos:exactMatch rdf:resource="http://viaf.org/processed/NLA%7C000035052711" /><skos:exactMatch rdf:resource="http://viaf.org/processed/NLIlat%7C000041704" /><skos:exactMatch rdf:resource="http://dbpedia.org/page/Wikipedia:WikiProject_Bob_Dylan" /><skos:exactMatch rdf:resource="http://viaf.org/processed/LAC%7C0008D6165" /><skos:exactMatch rdf:resource="http://viaf.org/processed/PTBNP%7C1270067" /><rdf:type rdf:resource="http://www.w3.org/2004/02/skos/core#Concept" /><dcterms:modified rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2010-02-22T06:44:10+00:00</dcterms:modified><dcterms:type>person</dcterms:type><dcterms:identitifer>46946176</dcterms:identitifer><dcterms:created rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2009-03-03T12:03:19+00:00</dcterms:created></rdf:Description></umbel:isAbout><rdf:type><rdf:Description rdf:about="http://xmlns.com/foaf/0.1/Person"></rdf:Description></rdf:type></rdf:Description></dcterms:creator><dcterms:language><rdf:Description rdf:about="http://purl.org/NET/marccodes/languages/eng#lang"></rdf:Description></dcterms:language><dcterms:subject><rdf:Description rdf:about="http://id.loc.gov/authorities/sh87003307#concept"><owl:sameAs><rdf:Description rdf:about="info:lc/authorities/sh87003307"></rdf:Description></owl:sameAs><skos:inScheme><rdf:Description rdf:about="http://id.loc.gov/authorities#topicalTerms"></rdf:Description></skos:inScheme><skos:inScheme><rdf:Description rdf:about="http://id.loc.gov/authorities#conceptScheme"></rdf:Description></skos:inScheme><skos:prefLabel>Popular music--1961-1970</skos:prefLabel><rdf:type><rdf:Description rdf:about="http://www.w3.org/2004/02/skos/core#Concept"></rdf:Description></rdf:type><dcterms:modified rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">1987-07-14T11:37:03-04:00</dcterms:modified><dcterms:created rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">1987-05-22T00:00:00-04:00</dcterms:created></rdf:Description></dcterms:subject><dcterms:subject><rdf:Description rdf:about="http://id.loc.gov/authorities/sh87003285#concept"><owl:sameAs><rdf:Description rdf:about="info:lc/authorities/sh87003285"></rdf:Description></owl:sameAs><skos:inScheme><rdf:Description rdf:about="http://id.loc.gov/authorities#topicalTerms"></rdf:Description></skos:inScheme><skos:inScheme><rdf:Description rdf:about="http://id.loc.gov/authorities#conceptScheme"></rdf:Description></skos:inScheme><skos:prefLabel>Blues (Music)--1961-1970</skos:prefLabel><rdf:type><rdf:Description rdf:about="http://www.w3.org/2004/02/skos/core#Concept"></rdf:Description></rdf:type><dcterms:modified rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">1987-07-14T16:41:41-04:00</dcterms:modified><dcterms:created rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">1987-05-22T00:00:00-04:00</dcterms:created></rdf:Description></dcterms:subject></rdf:Description></rdf:RDF>
* Connection #0 to host lccn.lcsubjects.org left intact
* Closing connection #0

The important thing to note here is the HTTP status code sent back. LinkedLCCN sends back a 206, partial content because it wants the agent to try again later. “Thank you for waiting, here is some data to get you started. If you come back, it’s possible I might have some more.”

And, indeed, if the agent came back:
$ curl -v -H “accept: application/rdf+xml” http://lccn.lcsubjects.org/93707283#i

* About to connect() to lccn.lcsubjects.org port 80 (#0)
*   Trying 208.83.140.6... connected
* Connected to lccn.lcsubjects.org (208.83.140.6) port 80 (#0)
> GET /93707283#i HTTP/1.1
> User-Agent: curl/7.19.6 (i386-apple-darwin9.8.0) libcurl/7.19.6 zlib/1.2.3
> Host: lccn.lcsubjects.org
> accept: application/rdf+xml
>
< HTTP/1.1 200 OK
< Date: Fri, 12 Mar 2010 20:26:18 GMT
< Server: Apache/2.2.12 (Ubuntu)
< X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.11
< Content-Length: 40807
< Status: 200
< Content-Type: application/rdf+xml
<
<rdf:RDF xmlns:n0="http://dbtune.org/musicbrainz/resource/vocab/" xmlns:mo="http://purl.org/ontology/mo/" xmlns:dcterms="http://purl.org/dc/terms/" xmlns:n1="http://www.holygoat.co.uk/owl/redwood/0.1/tags/" xmlns:n2="http://purl.org/vocab/bio/0.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:foaf="http://xmlns.com/foaf/0.1/" xmlns:skos="http://www.w3.org/2004/02/skos/core#" xmlns:bibo="http://purl.org/ontology/bibo/" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:owl="http://www.w3.org/2002/07/owl#" xmlns:umbel="http://umbel.org/umbel#" xmlns:rda="http://RDVocab.info/Elements/"><rdf:Description rdf:about="http://purl.org/NET/lccn/93707283#i"><mo:label><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/label/011d1192-6f65-45bd-85c4-0400dd45693e"><n0:label_sortname>Columbia Records</n0:label_sortname><n0:label_name>Columbia Records</n0:label_name><n0:label_labelcode>162</n0:label_labelcode><n0:tag_count>1</n0:tag_count><n0:label_type>4</n0:label_type><n0:alias>Columbia Phonograph Company</n0:alias><n0:alias>Columbia</n0:alias><n0:alias>Colombia Records</n0:alias><n0:alias>Columbia d (Sony BMG)</n0:alias><n0:alias>Columbia Records</n0:alias><n0:alias>Columbia US</n0:alias><rdfs:label>Columbia Records</rdfs:label><n1:taggedWithTag><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/tag/20"></rdf:Description></n1:taggedWithTag><n1:taggedWithTag><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/tag/748"></rdf:Description></n1:taggedWithTag><n1:taggedWithTag><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/tag/1584"></rdf:Description></n1:taggedWithTag><n1:taggedWithTag><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/tag/111"></rdf:Description></n1:taggedWithTag><n1:taggedWithTag><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/tag/273"></rdf:Description></n1:taggedWithTag><n1:taggedWithTag><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/tag/343"></rdf:Description></n1:taggedWithTag><n1:taggedWithTag><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/tag/7179"></rdf:Description></n1:taggedWithTag><n1:taggedWithTag><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/tag/284"></rdf:Description></n1:taggedWithTag><n2:event><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/label/011d1192-6f65-45bd-85c4-0400dd45693e/birth"></rdf:Description></n2:event><foaf:based_near><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/country/US"></rdf:Description></foaf:based_near><dc:description>1931-1990: only USA, Canada&amp;Japan. 1990 to present: worldwide</dc:description><rdf:type><rdf:Description rdf:about="http://purl.org/ontology/mo/Label"></rdf:Description></rdf:type></rdf:Description></mo:label><mo:catalogue_number>CS 8786</mo:catalogue_number><mo:track><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/track/5857b092-93ae-434d-b3f1-3f959396732b"></rdf:Description></mo:track><mo:track><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/track/98846b10-8951-43bc-ab24-c960e330cec8"></rdf:Description></mo:track><mo:track><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/track/88621637-8f03-427e-855f-4f52f712e80e"></rdf:Description></mo:track><mo:track><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/track/6d2b3714-478f-4fb1-9dfb-4a70c266453e"></rdf:Description></mo:track><mo:track><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/track/b654f8ad-071a-41c1-a1f7-1134de178ee8"></rdf:Description></mo:track><mo:track><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/track/9b08a5da-da77-4df8-b3ad-dcb481959013"></rdf:Description></mo:track><mo:track><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/track/f96f9b50-959d-4ef0-adc0-2995d179e6c8"></rdf:Description></mo:track><mo:track><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/track/2160818f-ce54-4502-a480-535389abef61"></rdf:Description></mo:track><mo:track><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/track/2f528602-ea36-480a-b1df-f7a5af36598e"></rdf:Description></mo:track><mo:track><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/track/19f571b4-b396-4113-9858-ab032074a3c7"></rdf:Description></mo:track><mo:track><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/track/00e37446-2e4c-409a-a8a1-ed94f1b01a57"></rdf:Description></mo:track><mo:track><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/track/536247b1-a87a-40c7-93e8-dd02fe3f3d54"></rdf:Description></mo:track><mo:track><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/track/67a68273-fa05-4ecf-aa85-648868a91b01"></rdf:Description></mo:track><rda:placeOfPublication><rdf:Description rdf:about="http://purl.org/NET/marccodes/countries/nyu#location"></rdf:Description></rda:placeOfPublication><rda:titleProper>The freewheelin' Bob Dylan</rda:titleProper><foaf:isPrimaryTopicOf><rdf:Description rdf:about="http://lccn.loc.gov/93707283"></rdf:Description></foaf:isPrimaryTopicOf><bibo:uri>http://hdl.loc.gov/loc.mbrsrs/lp0001.dyln</bibo:uri><bibo:lccn>93707283</bibo:lccn><rdf:type><rdf:Description rdf:about="http://purl.org/ontology/mo/Recording"></rdf:Description></rdf:type><dcterms:title>The freewheelin' Bob Dylan</dcterms:title><dcterms:creator><rdf:Description rdf:about="http://purl.org/NET/lccn/people/n50030190#i"><owl:sameAs><rdf:Description rdf:about="http://dbpedia.org/resource/Bob_Dylan"></rdf:Description></owl:sameAs><foaf:name>Dylan, Bob, 1941-</foaf:name><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r64001976#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007568523#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700813#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700816#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/unk84175336#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/95769390#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/99567433#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2002560219#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2002603175#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/72343809#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/78762560#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700762#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700760#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r64001986#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2001036739#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700847#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/76762320#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93727595#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93726950#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/00536083#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/unk84158135#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93727467#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2003636870#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93723725#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/88753098#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2008640899#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700841#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r65000579#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93705362#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700812#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700817#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007642887#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93711197#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700805#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700771#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700763#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/72762075#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2002042823#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700818#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700774#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/72763265#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/00717921#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/72761611#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93707558#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2002578760#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2010616561#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r68000463#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007657614#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/00725480#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/95776698#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2004304721#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/77760065#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700843#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700775#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/94771432#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700807#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r65001916#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/91759862#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2004593548#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/74217252#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/68128332#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700811#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2003696014#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/91762191#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700770#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007657624#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/71224759#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/71763700#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/74760392#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007659104#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93726958#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/98028826#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/74760100#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700748#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/unk84196471#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93712912#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/94023183#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700777#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/91759844#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/91759501#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/74762095#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2005045677#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93712878#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2009602196#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/85161136#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/99580439#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700758#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2001536938#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93703756#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700751#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700806#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700756#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700819#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/90750957#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/95701312#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2005434462#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/unk84219598#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/95786124#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93703149#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/99573938#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/79761946#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2002572371#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700804#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/66041425#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93710188#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/78056239#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93712977#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r67001398#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700244#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/66025502#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/86463558#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700815#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700768#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93727732#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/72002339#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93703465#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700844#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2005560717#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700753#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/91759880#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2008643295#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/unk84085959#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/76013692#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700800#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/92776389#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93711016#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93851416#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007657612#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/95129257#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700801#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700840#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700761#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/76760788#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007656198#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/70761002#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2003385743#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700765#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700755#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2008655199#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/73762412#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700242#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/92755348#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r64000162#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700846#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/77275475#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700769#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/80760343#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/97751545#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2006656398#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2008038164#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93715284#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700837#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700764#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700752#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700759#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2006657054#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/94770754#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93712995#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/00584949#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2004590181#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93729841#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/00717773#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2004592553#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2010616562#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2008300626#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/79761354#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93707283#i"><mo:label rdf:resource="http://dbtune.org/musicbrainz/resource/label/011d1192-6f65-45bd-85c4-0400dd45693e" /><mo:catalogue_number>CS 8786</mo:catalogue_number><mo:track rdf:resource="http://dbtune.org/musicbrainz/resource/track/5857b092-93ae-434d-b3f1-3f959396732b" /><mo:track rdf:resource="http://dbtune.org/musicbrainz/resource/track/98846b10-8951-43bc-ab24-c960e330cec8" /><mo:track rdf:resource="http://dbtune.org/musicbrainz/resource/track/88621637-8f03-427e-855f-4f52f712e80e" /><mo:track rdf:resource="http://dbtune.org/musicbrainz/resource/track/6d2b3714-478f-4fb1-9dfb-4a70c266453e" /><mo:track rdf:resource="http://dbtune.org/musicbrainz/resource/track/b654f8ad-071a-41c1-a1f7-1134de178ee8" /><mo:track rdf:resource="http://dbtune.org/musicbrainz/resource/track/9b08a5da-da77-4df8-b3ad-dcb481959013" /><mo:track rdf:resource="http://dbtune.org/musicbrainz/resource/track/f96f9b50-959d-4ef0-adc0-2995d179e6c8" /><mo:track rdf:resource="http://dbtune.org/musicbrainz/resource/track/2160818f-ce54-4502-a480-535389abef61" /><mo:track rdf:resource="http://dbtune.org/musicbrainz/resource/track/2f528602-ea36-480a-b1df-f7a5af36598e" /><mo:track rdf:resource="http://dbtune.org/musicbrainz/resource/track/19f571b4-b396-4113-9858-ab032074a3c7" /><mo:track rdf:resource="http://dbtune.org/musicbrainz/resource/track/00e37446-2e4c-409a-a8a1-ed94f1b01a57" /><mo:track rdf:resource="http://dbtune.org/musicbrainz/resource/track/536247b1-a87a-40c7-93e8-dd02fe3f3d54" /><mo:track rdf:resource="http://dbtune.org/musicbrainz/resource/track/67a68273-fa05-4ecf-aa85-648868a91b01" /><rda:placeOfPublication rdf:resource="http://purl.org/NET/marccodes/countries/nyu#location" /><rda:titleProper>The freewheelin' Bob Dylan</rda:titleProper><foaf:isPrimaryTopicOf rdf:resource="http://lccn.loc.gov/93707283" /><bibo:uri>http://hdl.loc.gov/loc.mbrsrs/lp0001.dyln</bibo:uri><bibo:lccn>93707283</bibo:lccn><rdf:type rdf:resource="http://purl.org/ontology/mo/Recording" /><dcterms:title>The freewheelin' Bob Dylan</dcterms:title><dcterms:creator rdf:resource="http://purl.org/NET/lccn/people/n50030190#i" /><dcterms:language rdf:resource="http://purl.org/NET/marccodes/languages/eng#lang" /><dcterms:subject rdf:resource="http://id.loc.gov/authorities/sh87003307#concept" /><dcterms:subject rdf:resource="http://id.loc.gov/authorities/sh87003285#concept" /><dcterms:isVersionOf rdf:resource="http://dbtune.org/musicbrainz/resource/record/942be4b0-12a2-4264-93a3-b45fa94c95c0" /></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/99583697#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93724609#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/75766013#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/66041424#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2008273503#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/96789283#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700776#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700820#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93711987#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700839#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2004585875#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/94162315#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700808#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/80772269#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r68000263#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700754#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2010616563#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2004577731#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/94770756#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2004571741#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/97109960#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r62000368#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/76353124#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93711249#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93037274#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/86753728#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/72760404#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700749#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700772#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/77018965#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700809#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/75762080#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r67001399#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2003643362#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/78531019#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2010617063#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93701439#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r68000262#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2003577486#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700766#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/76762851#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/unk84126692#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/94762678#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007657649#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700243#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93728350#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/92750713#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/94762887#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/95769054#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/72373613#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/99571326#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/91755140#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/92774846#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/77761257#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93711137#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2003643131#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2009015567#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/76770532#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/91761855#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93721323#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700803#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700810#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2010615421#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/94746592#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/72373606#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/92757783#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2002603171#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93712772#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700767#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2004056454#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r65000580#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/91759726#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93712861#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/81047774#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2005048013#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93702899#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/unk84096809#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700245#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/85040408#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/92754060#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/66052234#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/72251264#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700757#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700845#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/91761627#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700814#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2002556777#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700750#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/87754788#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/94770755#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700773#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/91760389#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r62000369#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2004560796#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/94750639#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2006571042#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r65001917#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/92754813#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700802#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2003574417#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93707133#i"></rdf:Description></foaf:made><foaf:made>
<rdf:Description rdf:about="http://purl.org/NET/lccn/2007700842#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/95769314#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r67001260#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2003573910#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/87752098#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2004462312#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/r64000161#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/unk85066726#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/71763246#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/99567232#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2006530396#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2001545149#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/91761930#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/2007700799#i"></rdf:Description></foaf:made><foaf:made><rdf:Description rdf:about="http://purl.org/NET/lccn/93709800#i"></rdf:Description></foaf:made><umbel:isAbout><rdf:Description rdf:about="http://viaf.org/viaf/46946176.rwo"></rdf:Description></umbel:isAbout><rdf:type><rdf:Description rdf:about="http://xmlns.com/foaf/0.1/Person"></rdf:Description></rdf:type></rdf:Description></dcterms:creator><dcterms:language><rdf:Description rdf:about="http://purl.org/NET/marccodes/languages/eng#lang"></rdf:Description></dcterms:language><dcterms:subject><rdf:Description rdf:about="http://id.loc.gov/authorities/sh87003307#concept"><owl:sameAs><rdf:Description rdf:about="info:lc/authorities/sh87003307"></rdf:Description></owl:sameAs><skos:inScheme><rdf:Description rdf:about="http://id.loc.gov/authorities#conceptScheme"></rdf:Description></skos:inScheme><skos:inScheme><rdf:Description rdf:about="http://id.loc.gov/authorities#topicalTerms"></rdf:Description></skos:inScheme><skos:prefLabel>Popular music--1961-1970</skos:prefLabel><rdf:type><rdf:Description rdf:about="http://www.w3.org/2004/02/skos/core#Concept"></rdf:Description></rdf:type><dcterms:modified rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">1987-07-14T11:37:03-04:00</dcterms:modified><dcterms:created rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">1987-05-22T00:00:00-04:00</dcterms:created></rdf:Description></dcterms:subject><dcterms:subject><rdf:Description rdf:about="http://id.loc.gov/authorities/sh87003285#concept"><owl:sameAs><rdf:Description rdf:about="info:lc/authorities/sh87003285"></rdf:Description></owl:sameAs><skos:inScheme><rdf:Description rdf:about="http://id.loc.gov/authorities#conceptScheme"></rdf:Description></skos:inScheme><skos:inScheme><rdf:Description rdf:about="http://id.loc.gov/authorities#topicalTerms"></rdf:Description></skos:inScheme><skos:prefLabel>Blues (Music)--1961-1970</skos:prefLabel><rdf:type><rdf:Description rdf:about="http://www.w3.org/2004/02/skos/core#Concept"></rdf:Description></rdf:type><dcterms:modified rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">1987-07-14T16:41:41-04:00</dcterms:modified><dcterms:created rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">1987-05-22T00:00:00-04:00</dcterms:created></rdf:Description></dcterms:subject><dcterms:isVersionOf><rdf:Description rdf:about="http://dbtune.org/musicbrainz/resource/record/942be4b0-12a2-4264-93a3-b45fa94c95c0"></rdf:Description></dcterms:isVersionOf></rdf:Description></rdf:RDF>
* Connection #0 to host lccn.lcsubjects.org left intact
* Closing connection #0

There is a much richer graph waiting for it.

Now, I have no idea if this is a valid application of the 206 response. The only references I see to it on the web deal with either cache proxies or range requests, but this seems like best way to alert the client that they aren’t getting the entire graph on this request.

So try it out and enjoy the new (and very basic) HTML interface.

Any comments, suggestions or criticisms on this approach are extremely welcome.

One of the byproducts of the “Communicat” work I had done at Georgia Tech was a variant of Ed Summersruby-marc that went into more explicit detail regarding the contents inside the MARC record (as opposed to ruby-marc which focuses on its structure).  It had been living for the last couple of years as a branch within ruby-marc, but this was never a particularly ideal approach.  These enhancements were sort of out of scope for ruby-marc as a general MARC parser/writer, so it’s not as if this branch was ever going to see the light of day as trunk.  As a result, it was a massive pain in the butt for me to use locally:  I couldn’t easily add it as a gem (since it would have replaced the real ruby-marc, which I use far too much to live without) which meant that I would have to explicitly include it in whatever projects I wanted to use it in and update any paths included accordingly.

So as I found myself, yet again, copying the TypedRecords directory into another local project (this one to map MARC records to RDF), I decided it was time to make this its own project.

One of the amazingly wonderful aspects of Ruby is the notion of “opening up an object or class”.  For those not familiar with Ruby, the language allows you to take basically any object or class, redefine it and add your own attributes, methods, etc.  So if you feel that there is some particular functionality missing from a given Ruby object, you can just redefine it, adding or overriding the existing methods, without having to reimplement the entire thing.  So, for example:

class String
  def shout
    "#{self.upcase}!!!!"
  end
end

str = "Hello World"
str.shout
=> "HELLO WORLD!!!!"

And just like that, your String objects gained the ability to get a little louder and a little more obnoxious.

So rather than design the typed records concept as a replacement for ruby-marc, it made more sense to treat it more as an extension to ruby-marc.  By monkey patching, the regular marc parser/writer can remain the same, but if you want to look a little more closely at the contents, it will override the behavior of the original classes and objects and add a whole bunch of new functionality.  For MARC records, it’s analogous to how Facets adds all kinds of convenience methods to String, Fixnum, Array, etc.

So, now it has its own github project:  enhanced-marc.

If you want to install it:

  gem sources -a http://gems.github.com
  sudo gem install rsinger-enhanced_marc

There’s some really simple usage instructions on the project page and I’ll try to get the rdocs together as soon as I can.  In a nutshell it works almost just like ruby-marc does:

require 'enhanced_marc'

records = []
reader = MARC::Reader.open('marc.dat')
reader.each do | record
  records << record
end

As it parses each record, it examines the leader to determine what kind of record it is:

  • MARC::BookRecord
  • MARC::SerialRecord
  • MARC::MapRecord
  • MARC::ScoreRecord
  • MARC::SoundRecord
  • MARC::VisualRecord
  • MARC::MixedRecord

and adds a bunch of format specific methods appropriate for, say, a map.

It’s possible to then simply extract either the MARC codes or the (English) human readable string that the MARC code represents:

record.class
=> MARC::SerialRecord
record.frequency
=> "d"
record.frequency(true)
=> "Daily"
record.serial_type(true)
=> "Newspaper"
record.is_conference?
=> false

or, say:

record.class
=> MARC::VisualRecord
record.is_govdoc?
=> true
record.audience_level
=> "j"
record.material_type(true)
=> "Videorecording"
record.technique(true)
=> "Animation"

And so on.

There is still quite a bit I still need to add.  It pretty much ignores mixed records at the moment.  It’s something I’ll need to eventually get to, but these are uncommon enough that it’s currently a lower priority.  I also need to provide some methods that evaluate the 007 field.  I haven’t gotten to this yet, just because it’s just a ton of tedium.  It would be useful, though, so I want to get it in there.

If there is interest, it could perhaps be extended to include authority records or holdings records.  It would also be handy to have convenience methods on the data fields:

record.isbn
=> "0977616630"
record.control_number
=> "793456"

Anyway, hopefully somebody might find this to be useful.

For the last couple of weeks I’ve returned to working on Alto Jangle connector, at least part-time.  I had shelved development on it for a while; I had a hard time finding anybody interested in using it and had reached a point where the development database I was working against was making it difficult to know what to expect in a real, live Alto system.  After I got wind of a couple of libraries that might be interested in it, I thought I should at least get it in a usable state.

One of the things that was vexing me prior to my hiatus was how to get Sybase to page through results in a semi-performant way.  I had originally blamed it on Grails, then when I played around with refactoring the connector in PHP (using Quercus, which is pretty slick by the way, to provide Sybase access via JDBC — the easiest way to do it) I realized that paging is just outside of Sybase’s capabilities.

And when you’re so used to MySQL, PostgreSQL and SQLite, this sort of makes your jaw drop (although, in its defense, it appears that this isn’t all that easy in Oracle, either — however, it’s at least possible in Oracle).

There seem to be two ways to do something like getting rows 375,000 – 375,099 from all of the rows in a table:

  1. Use cursors
  2. use SET ROWCOUNT 375100 and loop through and throw out the first 375,000 results.

The first option isn’t really viable.  You need write access to the database and it’s unclear how to make this work in most database abstraction libraries.  I don’t actually know that cursors do anything differently than option 2 besides pushing the looping to the database engine itself.  I was actually using cursors in my first experiments in JRuby using java.sql directly, but since I wasn’t aware of this problem at the time, I didn’t check to see how well it performed.

Option 2 is a mess, but this appears to be how GORM/Hibernate deals with paging in Sybase.  Cursors aren’t available in Quercus’ version of PDO, so it was how I had to deal with paging in my PHP prototypes, as well.  When I realized that PHP was not going to be any faster than Grails, I decided to just stick with Grails (“regular C-PHP” is out — compiling in Sybase support is far too heavy a burden).

This paging thing still needed to be addressed.  Offsets of 400,000 and more were taking more than twelve minutes to return.  How much more, I don’t know — I killed the request at the 12 minute mark.  While some of this might be result of a bad or missing index, any way you cut it, it wasn’t going to be acceptable.

I was kicking around the idea of exporting the “models” of the Jangle entities into a local HSQLDB (or whatever) mirror and then working the paging off of that.  I couldn’t help but think that this was sort of a waste, though — exporting from one RDBMS to another solely for the benefit of paging.  You’d have to keep them in sync somehow and still refer to the original Sybase DB for things like relationships and current item or borrower status.  For somebody that’s generally pretty satisfied with hacking together kludgy solutions to problems, this seemed a little too hack-y… even for my standards.

Instead, I settled on a different solution that could potentially bring a bunch of other features along with it.  Searchable is a Grails plugin for Compass, a project to easily integrate Lucene indexes with your Java domain classes (this would be analogous to Rails’ act_as_ferret).  When your Grails application starts up, Searchable will begin to index whatever models you declared as, well,  searchable.  You can even set options to store all of your attributes, even if they’re not actual database fields, alleviating the need to hit the original database at all, which is nice.  Initial indexing doesn’t take long — our “problem” table that took twelve minutes to respond takes less than five minutes to fully index.  It would probably take considerably less than that if the data was consistent (some of the methods to set the attributes can be pretty slow if the data is wonky — it tries multiple paths to find the actual values of the attribute).

What this then affords us is consistent access times, regardless of the size of the offset:  the 4,000th page is as fast as the second:  between 2.5 and 3.5 seconds (our development database server is extremely underpowered and I access it via the VPN — my guess is that a real, live situation would be much faster).

The first page is a bit slower.  I can’t use the Lucene index for the first page of results because there’s no way for Searchable to know if the WORKS_META table has changed since the last request since these changes wouldn’t be happening through Grails.  Since performance for the first hundred rows out of Sybase isn’t bad, the connector just uses it for the first page, then syncs the Lucene index with the database at the end of the request.  Each additional page then pulls from Lucene.  Since these pages wouldn’t exist until after the Lucene index is created and the Lucene index is recreated every time the Grails app is started, I added a controller method that checks the count of the Sybase table and the count of the Lucene index to confirm that they’re in sync (it’s worth noting that if the Lucene index has already been created once, this will be available right away after Grails starts — the reindexing is still happening, but in a temp location that will be moved to the default location once it’s complete overwriting the old index).

The side benefit to using Searchable is that it will make adding search functionality to Alto connector that much easier.  Building SQL statements from the CQL queries in the OpenBiblio connector was a complete pain the butt.  CQL to Lucene syntax should be considerably easier.  It seems like  it would be possible for these Lucene indexes to potentially alleviate the need for the bundles Zebra index that comes with Alto, eventually, but that’s just me talking, not any sort of strategic goal.

Anyway, thanks to Lucene, Sybase is behaving mostly like a modern RDBMS, which is a refreshing change.

While what I’m posting here might be incredibly obvious to anyone that understands unicode or Ruby better than me, it was new to me and might be new to you, so I’ll share.

Since Ed already let the cat out of the bag about LCSubjects.org, I can explain the backstory here.  At lcsh.info, Ed made the entire dataset available as N-Triples, so just before he yanked the site, I grabbed the data and have been holding onto it since.  I wrote a simple little N-Triples parser in Ruby to rewrite some of the data before I loaded it into the platform store I have.  My first pass at this was really buggy, I wasn’t parsing N-Triple literals well at all and was leaving out quoted text within the literal and whatnot.  I also, inadvertantly, was completely ignoring the escaped unicode within the literals and sending them verbatim.

N-Triples escapes unicode the same way Python string literals do (or at least this is how I’ve understood it), so 7⁰03ʹ43ʺN 151⁰56ʹ25ʺE is serialized into nt like: 7\\u207003\\u02B943\\u02BAN 151\\u207056\\u02B925\\u02BAE.  Try as I might, I could not figure out how to turn that back into unicode.

Jonathan Rochkind recommended that I look at the Ruby JSON library for some guidance, since JSON also encodes this way.  With that, I took a peek in JSON::Pure::Parser and modified parse_string for my needs.  So, if you have escaped unicode strings like this, and want them to be unicode, here’s a simple class to handle it.

$KCODE = 'u'
require 'strscan'
require 'iconv'
require 'jcode'
class UTF8Parser < StringScanner
  STRING = /(([\x0-\x1f]|[\\\/bfnrt]|\\u[0-9a-fA-F]{4}|[\x20-\xff])*)/nx
  UNPARSED = Object.new
  UNESCAPE_MAP = Hash.new { |h, k| h[k] = k.chr }
  UNESCAPE_MAP.update({
    ?"  => '"',
    ?\\ => '\\',
    ?/  => '/',
    ?b  => "\b",
    ?f  => "\f",
    ?n  => "\n",
    ?r  => "\r",
    ?t  => "\t",
    ?u  => nil,
  })
  UTF16toUTF8 = Iconv.new('utf-8', 'utf-16be')
  def initialize(str)
    super(str)
    @string = str
  end
  def parse_string
    if scan(STRING)
      return '' if self[1].empty?
      string = self[1].gsub(%r((?:\\[\\bfnrt"/]|(?:\\u(?:[A-Fa-f\d]{4}))+|\\[\x20-\xff]))n) do |c|
        if u = UNESCAPE_MAP[$&[1]]
          u
        else # \uXXXX
          bytes = ''
          i = 0
          while c[6 * i] == ?\\ && c[6 * i + 1] == ?u
            bytes << c[6 * i + 2, 2].to_i(16) << c[6 * i + 4, 2].to_i(16)
            i += 1
          end
          UTF16toUTF8.iconv(bytes)
        end
      end
      if string.respond_to?(:force_encoding)
        string.force_encoding(Encoding::UTF_8)
      end
      string
    else
      UNPARSED
    end
  rescue Iconv::Failure => e
    raise GeneratorError, "Caught #{e.class}: #{e}"
  end
end

I am not a programmer.

Since I first began writing code, my approach to learning a new language has been to take something that does the sort of thing I am looking for and start fiddling, seeing the results of the fiddling (most likely through some sort of error message) and refiddle until I start seeing the outcome I was looking for.  In mechanical terms, I open the hood, get out my biggest wrench and start whacking at things until the noises stop.  Or, in this case, start.

The arc of languages I primarily worked in at any given time is a pretty good reflection of this approach:  Perl, TCL, PHP, then Ruby with a small foray into Python.  All dynamic, all extremely whackable.  Where whacking doesn’t work, Google (or, previously, Yahoo or, previously, Alta Vista) generally does.  Cut, paste and resume whacking.

The same philosophy applies when it comes to developing new projects.  I know, basically, what I want on the other side, but I have absolutely no idea what it will take to get there.  Generally this means I’ll pick up the nearest tool on hand (usually a red colored wrench) and start whacking until I see what I want.  That the red wrench isn’t the right tool for the job isn’t the point, since I’m only looking for the destination, not the best route there (since I have no idea how to get there in the first place).  The more comfortable I get with a tool, the more likely I am to nest with it, since the detour of finding (and learning how to use) another tool slows me down from reaching the goal.

The perfect example of this was WAG the Dog.  PHP was a ridiculous language to try to use for it, but ping, ping, ping, ping, it worked!

So it stands to reason that I’ve never really taken to Java.  Java is not whacking.  Java is slowly, verbosely and deliberately constructing the specific parts you need to accomplish your goal.  Java is a toolbox full of parts and pieces I do not know the names of, what they do or how they would even do anything, much less the job I’m trying to accomplish.  Java is to my career what a Masters in Mechanical Engineering is to a wrench.  I don’t use Java because I don’t even know the questions to ask to get started in the right direction.

The irony is that when I was hired by Talis, I was ‘assigned’ (that’s a stronger term than really applies) to an entirely Java-based project, Keystone.  To this day, some 15 months later, I have contributed exactly 0.0 lines of code towards Keystone.

I am not a programmer.

However, I am a tinkerer.

In an effort to eat our own dogfood, I had begun to write a Jangle connector for our library management system, Alto.  Alto is built on Sybase and we already had a RESTful SOA interface, the aforementioned Keystone.  It would have been logical for me, were I a real programmer, to take Keystone, add some new models and routes and call it a connector.

But that’s not how I roll.

Initially, I took to using the JangleR Ruby framework to build the Alto connector, since all it would require is to query the appropriate tables and ping, ping, ping, ping things until JRuby and Warbler could give me a .war file.

Sybase, however, does not take well to whacking.  Especially from Ruby.  ActiveRecord-JDBC didn’t work.  Not sure if it was our particular schema or JDBC setup or just ActiveRecord, but no dice.  I couldn’t get Ribs to work at all, which is just as well, probably.  Finally, I had success just using java.sql objects directly in JRuby, but, since I really didn’t know what I was doing, I started worrying about connection pooling and leaving connections open and whatnot.  No need to show off that I have no idea by gobbling up all the resources on some customer’s Alto server.

At one point, on a lark, I decided to try out Grails, Groovy‘s web framework inspired by Rails, to see if I could have more luck interacting with Sybase.  My rationale was, “Keystone uses Hibernate, GORM (Grails’ ORM) uses Hibernate, maybe it will work for me!”.  And, it did.

So here I am, one week into using Groovy.  Just like I used Rails as an introduction to Ruby, Grails serves that purpose with Groovy pretty well.  I can’t say I love the language, but that’s purely my bias; anything that isn’t Ruby, well, isn’t Ruby.  I am certainly doing some thing inefficiently since I am learning the language as I go along.  The fact that there just isn’t much documentation (and the existing documentation isn’t all that great) doesn’t help.

For example, none of my GORM associations work.  I have no idea why.  It could very well be the legacy Sybase schema, or I might be doing something wrong in my Domain Model class.  I don’t have any idea and I don’t have any idea where to look either for an appropriate error or for a fix.  It’s not a huge issue, though, and so far I’ve just worked around it by writing methods that do roughly what I would have needed the associations to do.  Ping, ping, ping.

I also cannot make Services work the way they show in the documentation.  My controllers can’t find them when I do it like the docs, my domain models can’t find them when I do it like the doc…  But it’s no big deal.  I set my methods to be static, call the class directly, and everything works fine.  I’m not doing anything remotely sophisticated with them, so I can keep my hacks for now.

Being able to dynamically load Java classes, iterate over things with a call like foo.each { bar = it.baz } is pretty nice.  I am really amazed at what Groovy offers when it comes to working with Java classes, it’s like being able to audit those M.E. Master’s classes.  I am learning a considerable amount about Java by being able to whack away at objects and classes within the Groovy shell.

I’m not sure that Groovy was really intended for people like me, however.  All of the documentation and even some of the syntax seem to have the expectation that you are a Java developer looking for something dynamic.  It reminds me of a Perl developer going to PHP.  They are syntactically and functionally similar.  In many ways, a Perl developer will find PHP an easier language to use to get a simple, running web application.  And they always have the fallback of Perl, if they need it.  A Python developer that has to use PHP will probably curse a lot.  Groovy seems to have the same sort of relationship to Java.  A Java developer would probably immediately feel comfortable and find it amazingly easy to get up and running.  A Ruby developer (well, this Ruby developer) finds it a little more alien.

Groovy doesn’t have a tremendous amount of native language support for specific tasks, relying instead on the vast amount Java libraries out there to do, basically, anything.  This makes perfect sense and I don’t fault Groovy in the slightest for this choice, but relying on Java for functionality means factories and stream buffers and all the other things Java consists of.  Java developers would feel home.  I find it needs some getting used to.

Also needing to declare your variables.

And I’m sure I’m not really using closures to their fullest potential.

Overall, it’s nice to have this new addition to my toolbox.  Groovy is definitely whackable and development for the Jangle connector has been remarkably fast.  I expect the Aspire (née List) and Prism teams to have something to try out by the end of the month.  And for basically being Java, that ain’t bad.

When and if I rewrite the Alto connector, I’ll probably opt for GroovyRestlet over Grails, but I definitely couldn’t have gotten where I have at this point without the convention and community of Grails.  It’s a really good starting point.

Of course, none of this would have been necessary if it wasn’t for Sybase.  Consider this day one of my campaign to eventually migrate Talis Alto from Sybase to PostgreSQLPing, ping, ping.

Something I’ve taken it upon myself to do since I joined Talis is make ActiveRDF a viable client to access the Platform.  While this is mostly selfishness on my part (I want to keep developing in Ruby and there’s basically no RDF support right now, plus this gives me a chance to learn about the RDF/SPARQL-y aspects of the Platform), I also think that libraries like this can only help democratize the Platform.

So far, it’s been pretty ugly.  I haven’t had much time to work on it, granted, but the time I’ve spent on it has made me think that there will be a lot of work to do.  Couple this with some of the things that make the Platform difficult to work with in Ruby anyway (read: Digest Authentication) and this might be a more uphill battle than I’ll ever have time for, but I figure it’s either this or go back to Python and I’m not quite ready to give up on Ruby yet.

Currently, performance is abysmal with ActiveRDF against the Platform, so I’ll need to think of shortcuts to improve that (I’m not even considering write access presently).  Here’s some code (this is as much for my benefit, so I can remember what I’ve done) to work with Ian Davis’ Quotations Book Example store:

require ‘time’ # Otherwise ActiveRDF starts freaking out about DateTime
require ‘active_rdf’

$activerdf_without_xsdtype = true
# less than ideal, but without it, ActiveRDF sends
# ^^<http://www.w3.org/2001/XMLSchema#string> with string literals even if you don’t want
# to send the datatype.  I haven’t actually tried it with other datatypes to see how this breaks
# down the road.

ConnectionPool.set_data_source(:type => :sparql, :results => :sparql_xml, :engine=>:joseki,  :url=> “http://api.talis.com/stores/iand-dev2/services/sparql”)

Namespace.register :foaf, “http://xmlns.com/foaf/0.1/”
Namespace.register :dc, “http://purl.org/dc/elements/1.1/”
Namespace.register :quote, “http://purl.org/vocab/quotation/schema”

QUOTE::Quotations.find_by_dc::creator(“Loren, Sophia”).each do | quote |

# print the important stuff from each graph

# http://purl.org/vocab/quotation/schema#quote has to be manually added as a predicate
# the “#” seems to cause problems
quote.add_predicate(:quote, QUOTE::quote)
puts quote.quote
puts quote.subject
puts quote.rights
puts quote.isPrimaryTopicOf

end

If you actually try to execute this, you’ll see that it takes a long time to run (God help you if you try it on QUOTE::Quotations.find_by_dc::subject(“Age and Aging”)).  A really long time.

If you set some environment vars before you go into irb:

$ export ACTIVE_RDF_LOG_LEVEL=0
$ export ACTIVE_RDF_LOG=./activerdf.log

then you can tail -f activerdf.log and see what exactly is happening.

After ActiveRDF does it’s initial SPARQL query (SELECT DISTINCT ?s WHERE { ?s <http://purl.org/dc/elements/1.1/creator> “Loren, Sophia” . }), it’s doing two things for every request in the block:

  1. a SPARQL query for every predicate associated with the URI (http://api.talis.com/stores/iand-dev2/services/sparql?query=SELECT+DISTINCT+%3Fp+WHERE+%7B+%3Chttp%3A%2F%2Fapi.talis.com%2Fstores%2Fiand-dev2%2Fitems%2F1187139384317%3E+%3Fp+%3Fo+.+%7D+)
  2. a SPARQL query for the value of the attribute (predicate):  http://api.talis.com/stores/iand-dev2/services/sparql?query=SELECT+DISTINCT+%3Fo+WHERE+%7B+%3Chttp%3A%2F%2Fapi.talis.com%2Fstores%2Fiand-dev2%2Fitems%2F1187139384317%3E+%3Chttp%3A%2F%2Fpurl.org%2Fdc%2Felements%2F1.1%2Fcreator%3E+%3Fo+.+%7D

for every predicate in the graph.  You can imagine how crazily inefficient this is, since to get every value for a resource, you have to make a different HTTP request for each one.

Obviously this would be a lot easier if it used DESCRIBE rather than SELECT, but without a real RDF library to parse the resulting graph, I’m not sure how ActiveRDF would deal with what the triple store returned.

So, anyway, these are some of the hurdles in making ActiveRDF work with the Platform, but I’m not quite ready to throw in the towel, yet.

Sometime in November, I came to the realization that I had horribly misinterpreted the NISO Z39.88/OpenURL 1.0 spec.  I’m on the NISO Advisory Committee for OpenURL (which makes this even more embarrassing) and was reviewing the proposal for the Request Transfer Message Community Profile and its associated metadata formats when it dawned on me that my mental model was completely wrong.  For those of you that have primarily dealt with KEV based OpenURLs (which is 99% of all the OpenURLs in the wild), I would wager that your mental model is probably wrong, too.

A quick primer on OpenURL:

  • OpenURL is a standard for transporting ContextObjects (basically a reference to something, in practice, mostly bibliographic citations)
  • A ContextObject (CTX, for short from now on) is comprised of Entities that help define what it is.  Entities can be one of six kinds:
    • Referent – this is the meat of the CTX, what it’s about, what you’re trying to get context about.  A CTX must have one referent and only one.
    • ReferringEntitydefines the resource that cited the referent.  This is optional and can only appear once.
    • Referrer – the source of where the CTX came from (i.e. the A&I database).  This is optional and can only appear once.
    • Requester – this is information about who is making the request (i.e. the user’s IP address).  This is optional and can only appear once.
    • ServiceType – this defines what sorts of services are being requested about the referent (i.e. getFullText, document delivery services, etc.).  There can be zero or many ServiceType entities defined in the CTX.
    • Resolver these are messages specifically to the resolver about the request.  There can be zero or more Resolver entities defined in the CTX.
  • All entities are basically the same in what they can hold:
    • Identifiers (such as DOI or IP Address)
    • By-Value Metadata (the metadata is included in the Entity)
    • By-Reference Metadata (the Entity has a pointer to a URL where you can retrieve the metadata, rather than including it in the CTX itself)
    • Private Data (presumably data, possibly confidential, between the entity and the resolver)
  • A CTX can also contain administrative data, which defines the version of the ContextObject, a timestamp and an identifier for the CTX (all optional)
  • Community Profiles define valid configurations and constraints for a given use case (for instance, scholarly search services are defined differently than document delivery).  Context objects don’t actually specify any community profile they conform to.  This is a rather loose agreement between the resolver and the context object source:   if you provide me with a SAP1, SAP2 or Dublin Core compliant OpenURL, I can return something sensible.
  • There are currently two registered serializations for OpenURL:  Key/Encoded Values where all of the values are output on a single string, formatted as key=value and delimited by ampersands (this is what majority of all OpenURLs that currently exist look like) and XML (which is much rarer, but also much more powerful)
  • There is no standard OpenURL ‘response’ format.  Given the nature of OpenURL, it’s highly unlikely that one could be created that would meet all expected needs.  A better alternative would be for a particular community profile to define a response format since the scope would be more realistic and focused.

Looking back on this, I’m not sure how “quick” this is, but hopefully it can bootstrap those of you that have only cursory knowledge of OpenURL (or less).  Another interesting way to look at OpenURL is Jeff Young’s 6 questions approach, which breaks OpenURL down to “who”, “what”, “where”, “when”, “why” and “how”.

One of the great failings of OpenURL (in my mind, at least) is the complete and utter lack of documentation, examples, dialog or tutorials about its use or potential.  In fact, outside of COinS, maybe, there is no notion of “community” to help promote OpenURL or cultivate awareness or adoption.  To be fair, I am as guilty as anybody for this failure, since I had proposed making a community site for OpenURL, but due to a shift in job responsibilities and then the wholesale change in employers, coupled with the hacking of the server it was to live on, left this by the wayside.  I’m putting this back on my to do list.

What this lack of direction leads to is that would-be implementors wind up making a lot of assumptions about OpenURL.  The official spec published at NISO is a tough read and is generally discouraged by the “inner core” of the OpenURL universe (the Herbert van de Sompels, the Eric Hellmans, the Karen Coyles, etc.) in favor of the “Implementation Guidelines” documents.  However, only the KEV Guidelines are actually posted there.  The only other real avenue for trying to come to grips with OpenURL is to dissect the behavior of link resolvers.  Again, in almost every instance this means you’re working with KEVs and the downside of KEVs is that they give you a very naive view of OpenURL.

KEVs, by their very nature, are flat and expose next to nothing about the structure of the model of the context object they represent.  Take the following, for example:

url_ver=Z39.88-2004&url_tim=2003-04-11T10%3A09%3A15TZD
&url_ctx_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Actx&ctx_ver=Z39.88-2004
&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&ctx_id=10_8&ctx_tim=2003-04-11T10%3A08%3A30TZD
&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.aulast=Vergnaud
&rft.auinit=J.-R&rft.btitle=D%C3%A9pendances+et+niveaux+de+repr%C3%A9sentation+en+syntaxe
&rft.date=1985&rft.pub=Benjamins&rft.place=Amsterdam%2C+Philadelphia
&rfe_id=urn%3Aisbn%3A0262531283&rfe_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook
&rfe.genre=book&rfe.aulast=Chomsky&rfe.auinit=N&rfe.btitle=Minimalist+Program
&rfe.isbn=0262531283&rfe.date=1995&rfe.pub=The+MIT+Press&rfe.place=Cambridge%2C+Mass
&svc_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Asch_svc&svc.abstract=yes
&rfr_id=info%3Asid%2Febookco.com%3Abookreader

Ugly, I know, but bear with me for a moment.  From this example, let’s focus on the Referent:

rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.aulast=Vergnaud
&rft.auinit=J.-R&rft.btitle=D%C3%A9pendances+et+niveaux+de+repr%C3%A9sentation+en+syntaxe
&rft.date=1985&rft.pub=Benjamins&rft.place=Amsterdam%2C+Philadelphia

and then let’s make this a little more human readable:

rft_val_fmt:  info:ofi/fmt:kev:mtx:book
rft.genre:  book
rft.aulast:  Vergnaud
rft.auinit:  J.-R
rft.btitle:  Dépendances et niveaux de représentation en syntaxe
rft.date:  1985
rft.pub:  Benjamins
rft.place:  Amsterdam, Philadelphia

Looking at this example, it’s certainly easy to draw some conclusions about the referent, the most obvious being that it’s a book.

Actually (and this is where it gets complicated and I begin to look pedantic) it’s really only telling you, I am sending some by value metadata in the info:ofi/fmt:kev:mtx:book format, not that the thing is actually a book (although the info:ofi/fmt:kev:mtx:book metadata values do state that, but, ignore that for a minute since genre is optional).

The way this actually should be thought of:

ContextObject:
    Referent:
       Metadata by Value:
          Format:  info:ofi/fmt:kev:mtx:book
          Metadata:
             Genre:  book
             Btitle:  Dépendances et niveaux de représentation en syntaxe
             …
    ReferringEntity:
       Identifier:  urn:isbn:0262531283
       Metadata by Value:
           Format:  info:ofi/fmt:kev:mtx:book
           Metadata:
               Genre:  book
               Isbn:  0262531283
               Btitle:  Minimalist Progam
               …
    Referrer:
       Identifier:  info:sid/ebookco.com:bookreader
    ServiceType:
       Metadata By Value:
           Format:  info:ofi/fmt:kev:mtx:sch_svc
           Metadata:
               Abstract:  yes

So, this should still seem fairly straightforward, but the hierarchy certainly isn’t evident in the KEV.  It’s a good starting point to begin talking about the complexity of working with OpenURL, though, especially if you’re trying to create a service that consumes OpenURL context objects.

Back to the referent metadata.  The context object didn’t have to send the data in the “metadata by value” stanza.  It could have just sent the identifier “urn:isbn:9027231141” (and note in the above example, it didn’t have an identifier at all).  It could also have sent metadata in the Dublin Core format, MARC21, MODS, ONIX or all of the above (the Metadata By Value element is repeatable) if you wanted to make sure your referent could be parsed by the widest range of resolvers. While all of these are bibliographic formats, in Request Transfer Message context objects (which would be used for document delivery, which got me started down this whole path), you would conceivably have one or more of the aforementioned metadata types plus a Request Transfer Profile Referent type that describes the sorts of interlibrary loan-ish types of data that accompany the referent as well as an ISO Holdings Schema metadata element carrying the actual items a library has, their locations and status.

If you only have run across KEVs describing journal articles or books, this may come as a bit of a surprise.  Instead of saying the above referent is a book, it becomes important to say that the referent contains a metadata package (as Jonathan Rochkind calls it) that is in this (OpenURL specific) book format.  In this regard, OpenURL is similar to METS.  It wraps other metadata documents and defines the relationships between them.  It is completely ambivalent about the data it is transporting and makes no attempt to define it or format it in any way.  The Journal, Book, Patent and Dissertation formats were basically contrived to make compatibility with OpenURL 0.1 easier, but they are not directly associated with OpenURL and could have just as easily been replaced with, say, BibTex or RIS (although the fact that they were created alongside Z39.88 and are maintained by the same community makes the distinction difficult to see).

What this means, then, is that in order to know anything about a given entity, you also need to know about the metadata format that is being sent about it.  And since that metadata could literally be in any format, it means there are lot of variables that need to be addressed just to know what a thing is.

For the Umlaut, I wrote an OpenURL library for Ruby as a means to parse and create OpenURLs.  Needless to say, it was originally written with that naive, KEV-based, mental model (plus some other just completely errant assumptions about how context objects worked) and, because of this, I decided to completely rewrite it.  I am still in the process of this, but am struggling with some core architectural concepts and am throwing this out to the larger world as an appeal for ideas or advice.

Overall the design is pretty simple:  there is a ContextObject object that contains a hash of the administrative metadata and then attributes (referent, referrer, requester, etc.) that contain Entity objects.

The Entity object has arrays of identifiers, private data and metadata.

And then this is where I start to run aground.

The original (and current) plan was to populate the metadata array with native metadata objects that are generated by registering metadata classes in a MetadataFactory class.  The problem, you see, is that I don’t want to get into the business of having to create classes to parse and access every kind of metadata format that gets approved for Z39.88.  For example, Ed Summers’ ruby-marc has already solved the problem of effectively working with MARC in Ruby, so why do I want to reinvent that wheel?  The counter argument is, by delegating these responsibilities to third party libraries, there is no consistency of APIs between “metadata packages”.  A method used in format A may very well raise an exception (or, worse, overwrite data) in format B

There is a secondary problem that third party libraries aren’t going have any idea that they’re in an OpenURL context object or even know what that is.  This means there would have to be some class that handles functionality like xml serialization (since ruby-marc doesn’t know that Z39.88 refers to it as info:ofi/fmt:xml:xsd:MARC21), although this can be handled by the specific metadata factory class.  This would also be necessary when parsing an incoming OpenURL since, theoretically, every library could have a different syntax for importing XML, KEVs or whatever other serialization is devised in the future.

So I’m looking for advice on how to proceed.  All ideas welcome.

…although LinuX_Xploit_Crew, with all due respect, I think it actually is.

Oh well, we’re back with a new theme (which nobody will see except to read comments, since I’m pretty sure all traffic comes from the code4lib planet) and an updated WordPress install. Look out, world!

So, in the downtime here’s a non-comprehensive rundown of what I’ve been working on:

  1. I’ve written an improved (at least, I think it’s improved) alternative to Docutek’s Eres RSS interface. Frankly, Docutek’s sucked. Maybe we have an outdated version of Eres, but the RSS feeds would give errors because you have to click through a copyright scare page before you can view reserves, but you can’t link the RSS links to this form and get the item. I wrote a little Ruby/Camping app that takes urls like: http://eres.library.gatech.edu/course/WS-1001-A/Fall/2007 and turns that into a usable feed. I needed the course id/term/year format to show them in Sakai. My favorite part of this project was finding Rubyscript2exe. This allows me to just bundle one file (the compiled camping app) plus a configuration file. Granted, an asp.net app would be even easier for sites to install, but I didn’t have time to learn asp.net. I have more ideas of what I would like to do with this (such as show current circ status for physical reserves), but in the chaos that is our library reorg, I haven’t gotten around to even showing anybody what I’ve written so far.
  2. I broke ground on a Metalib X-Server Ruby library. It took me a while to wrap my head around how this needed to be modelled, but I think it’s starting to take shape. It doesn’t actually perform queries, yet, but it connects to the server, allows you to set the portal association and find and set the category to search. Quicksets and MySets are all derivations of the same concept, so I don’t think it will be take me long to actually incorporate actual searching. For proof-of-concept, I plan on embedding this library in MemoryHole, our Solr-based discovery app. I’ve actually stopped development of MemoryHole so we can focus on vuFind, since they do functionally the same thing and I’d rather help make vuFind better than replicate everything it does, only in Ruby. The reason I’m doing this proof-of-concept in MemoryHole rather than vuFind is solely due to familiarity and time.

In other news, my last post seems to have caused a bit of a stir. My plan is to write a response, but the short of it is that I feel the arguments for an MLS are extremely classist.

Also my bathroom is finished and it looks great.

ERESidue

Blogged with Flock

Tags:

One thing I hope that never happens at Access (and hope it hasn’t already happened and I’m too much of a super arrogant programmer to notice) is the “hotshot developers” gather round a table and whip out their laptops and break only to give supra-obtuse presentations on how to use their technologies (and opening by saying they aren’t going to explain anything to those that might not know the specifics of their particular technologies).

If Sakai really wants to become a ‘community driven software’, they really need to address this impenetrable geeky cult of personality.

Code4lib, of course, is already completely populated by super arrogant programmer types, so I’m not as worried about it.