Getting started with Semantic Technologies

Recently Ontotext launched a Self-Service Semantic Suite – shorted to S4. S4 provides a set of services for low-cost (currently free) on-demand text analytics and metadata management on the cloud. This provides a great way to get acquainted with Semantic Technologies.

Available S4 services

S4 currently offers the following services:

1. Text analytics for News content, Biomedical content or Twitter content
2. Linked Data server with reliable access to the DBpedia, FactForge, GeoNames, WordNet, MusicBrainz, and New York Times datasets 
3. Self-managed RDF database (GraphDB) on the cloud

Trying out Text analytics

Text analytics – in this context – is about finding out what is important in texts (natural language), and using this information.

To try this out, copy some text containing some Persons and Places, biomedical terms and/or Twitter content. Preferably in English for best results, but other languages will produce results as well.
Go to the S4 homepage and click on “Demo S4 today for free”. Paste your text in the Text Analytics box; choose whether your text is more News, Biomedical or Twitter oriented, and hit Execute.

Your result will show the provided text with different types of terms highlighted in different colours. See the below example.

Figure 1. Example of an annotated text

If you hover over an annotated term, it will show extra information. For instance for an organisation it will show the location in DBpedia (the semantic version of Wikipedia). In my example this makes clear that this article is not about some IMF, it is about the IMF, and more info is available on http://dbpedia.org/page/International_Monetary_Fund.

So what’s in it for you? 

This service can provide all kinds of structure and information on topics that can help you to classify, understand, link and enrich information.

Trying out Semantic queries

S4 also lets you try out semantic queries using SPARQL, the query language for semantically stored information such as the DBpedia.
Go to the S4 homepage and click on “Demo S4 today for free”. Go to the LOD Access Tab. Select a query from the Pulldown. Let’s try “Find airports near London”.
The SPARQL query is:

PREFIX geo-pos: <http://www.w3.org/2003/01/geo/wgs84_pos#>
PREFIX omgeo: <http://www.ontotext.com/owlim/geo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
PREFIX dbp-ont: <http://dbpedia.org/ontology/>
PREFIX ff: <http://factforge.net/>
PREFIX om: <http://www.ontotext.com/owlim/>

SELECT distinct ?airport ?label ?RR
WHERE {
        dbpedia:London geo-pos:lat ?latBase ;
                       geo-pos:long ?longBase .
        ?airport omgeo:nearby(?latBase ?longBase "50mi");
                 a dbp-ont:Airport ;
                 ff:preferredLabel ?label ;
                 om:hasRDFRank ?RR .
      } ORDER BY DESC(?RR)

Even without a SPARQL crash-course, this is quite easy to read:

• First some definitions are introduced
• The query will return the airport ID, name and some rank (RR)
• From DBpedia the latitude and longitude of London are retrieved
• Only results that have the DBpedia Ontology type “Airport” are selected, 
• They must be nearer than 50 miles to London, according to the Owlim Geospatial function “Nearby”

Click on “Execute” and have a look at the results. Try out some of the other queries as well.

So what’s in it for you? 

An enormous wealth of structured information is available for you to use. Were you aware that you could ask Wikipedia such detailed questions? Be aware: extensive knowledge is needed to write such concise statements and really use the results.

Next step

Now that you know that Text analysis and Semantic queries are available, what is your next step in using Semantic technologies?

Big improvements for Big Content

At HintTech we are one of the first able to test MarkLogic 8, our favorite Big Data & Big Content platform.

On MarkLogic World earlier this year, MarkLogic offered an insight in the new features, such as JavaScript & JSON support, SPARQL 1.1 & Inferencing and Bitemporal support.

In the coming days we will post blogs with revealing updates & inside information about MarkLogic 8 and our test drive of these features.

Read the Blogs through our corporate website, or subscribe there and we will keep you posted.

Be the first to know about MarkLogic 8!

MarkLogic World Amsterdam

May 15th, the MarkLogic World Tour touched down in Amsterdam. Approximately 300 attendees flocked the Amsterdam Arena to be immersed in MarkLogic and it possibilities. As longstanding MarkLogic partner and implementer, my employer Dayon and HintTech sponsored this event. MarkLogic presented interesting new Developments, Case studies and Technical Sessions.

Big Content challenges 

Keynote speaker an MarkLogic CEO Gary Bloom and Joe Pasqua, Senior Vice President of Product Strategy explained MarkLogic choices for addressing Big Data challenges, but also the Big Content challenges (see my earlier blog) such as Heterogenic structured data: supporting Multi Schema XML and changing schema's.

MarkLogic 8

MarkLogic 7 already introduced a lot of interesting new features; MarkLogic 8 raises the bar some more:

• JavaScript & JSON support
  Building Applications on MarkLogic will be available to lots more developers: front-end oriented developers and other developers who prefer {} over <>
• SPARQL 1.1 & Inferencing
  The Triple Store in MarkLogic 7 is a good starting point, but becomes much more complete by adding Inferencing and SPARQL 1.1 support
• Bitemporal support
  A very interesting feature, which I could use immediately: What did we know at a certain point in time?

Cases

I visited the more business oriented tracks, with cases from Standards Norway, Intel, and the Healthcare.gov case, known to us as Obamacare. MarkLogic explained that their parts of the system kept performing throughout the toughest time. If only all components would have been this reliable…
Some of our customers presented as well: Ted van Dongen of Swets told how a great tool needs good people to implement. Thanks for the compliments Ted!
Our (previously) own Michel de Ru contributed for Newz to a panel session on Sharing Data despite the Silos.

Can you have too much of a good thing?

I will admit to anyone that I am biased: I think MarkLogic is a great tool that solves many challenges my customers face. But I had an interesting discussion: with all these new features, will MarkLogic become a too monolithic solution? Can you have too much of a good thing?
My 2 eurocents are: For now, keep piling on the good stuff, MarkLogic!

Big Content challenges

At Dayon, we are used to work with Big Data. Coming from a publisher’s background, we have provided content solutions to publishers since 1997.

I read some stories about Big Content, and was intrigued that Gartner saw Big Content as the unstructured part of Big Data. To me, Big Content is the structured version of Big Data.

Let me explain this and address some challenges and Big Content technologies.

Planned Variety

In Terms of the three Big Data V’s (Volume, Velocity and Variety), publishers content is odd. Since the goal of publishers is to make a profit from providing content, content must be able to be published to a vast arrange of channels. To enable this, content must be structured (preferably in XML) and enriched with metadata. Any Variety is planned, because unplanned Variety leads to unplanned structures and/or unplanned publications.

Data is generated, where Content is handcrafted. Tweets en Facebook-posts are only lightly structured, but Blog posts are already quite structured. Some numbers by Chartbeat can be found here and a useful insight by Fastcompany on the rise of “Big Content” as a marketing Tool.

Publishers Content is usually completely structured: XML + Meta Data, sometimes already as RDF Triples (read my earlier Blog post on Semantic Technologies).

So to me, Content is structured Data. Big Content problems differ from other Big Data problems, where handling the Variety to understand your data is a big issue. Therefore, I would like to label the publishers challenge to be a Big Content challenge.

So how big is Big Content?

A quick scan at some of our publishing clients provided these numbers (XML only!):

1. Publisher 1: 10 million files, 25 GB
2. Publisher 2: 750.000  files, 15 GB
3. Publisher 3: 150 million files, 15 GB
4. Publisher 4: 1 million files, 15.000 new files per day (max)
5. Publisher 5: 45 million files, 20.000 new files per day (max)
6. Publisher 6: 500.000  files

Challenges of Big Content

With these numbers in mind, what are the challenges for Big Content?

1. Volume - XML: Are 30 million XML files a challenge? Or 25 GB in XML a challenge? It really should not be, but in reality I have met quite some technologies struggling with these amounts. An XML system should be true XML to handle this amount of data. XML isn't hard. Doing XML right is hard. If you don’t do XML right, 100.000 files or 1 GB of XML can get you plenty of headaches.
2. Volume - Other file types: Alas, not all Content is XML. Many Publishers still manage huge amounts of HTML, PDF or other file formats. With PDF, huge numbers often also turn into huge volumes because multi-channel and hence print-quality PDF is stored.If you have to index lots of other file types, do a proper intake process per file and weed out the corrupt and the largest files.
3. Volume - Subscriptions: At various clients I encountered the problem that Big Content is offered in large amounts of different Subscriptions. Whereas a large amounts of different Subscriptions are not a problem in itself, the combination of Big Content and Big (number of) Subscriptions often is. So if you offer lots of data, be smart about the number of Subscriptions.
4. Volume - Triples: Nearly all Publishers storing Big Content are looking into Triples as a way to store and link Meta Data from their XML files. Storing your Meta Data in a Triple Store, and Linking it to the Linked Open Data can be a very good idea, but this calls for a Big Triple Store. A set of 1 billion Triples isn't exceptional, but also requires Big Content Technology.
5. Velocity - Real Time Indexing: Failing at real time indexing is usually the first sign that you are becoming a Big Content publisher. Many technologies struggle with incremental updates, needing complete re-indexing, which in term leads to strange solutions such as overnight indexing, flip-flopping or indexes out of sync with the rest of the front-end.
6. Velocity - Real Time Alerting: The value of Content depends on its relevance, and timeliness is a huge factor in relevance. Real Time Alerting will offer a competitive edge to content users. To provide Real Time Alerting, XML store need to handle alerting efficiently (using minimal resources) at load time
7. Variety - Presentation: A Big Content challenge can be how to present all of this Content. If a simple “What’s New” view results in 20.000 hits, what are you going to show the customer?The most used solutions are:
1. Provide a Search Only interface
2. Provide as much structure from Meta Data as possible to assist the user in drilling down to the most useful Content
8. Variety - Enrichment: If the Meta Data you need to provide useful segmentation of your Big Content to your end users just isn't there; there is a need for additional Enrichment. Big Content will (due to costs) call for automated enrichment using Natural Language Processing

Big Content Technology

At Dayon / HintTech we strongly believe that Big Content challenges require specialized Big Content Technology. Here are some of the Big Content Technologies we have implemented:

1. MarkLogic
   Several of our Big Content clients have selected MarkLogic as their content platform. I believe that MarkLogic is the best XML store and indexer available at this moment.
   As a big bonus, MarkLogic comes with all kinds of useful features such as XQuery, an Application Server, and now even a Triple Store.
   Find out more about MarkLogic at MarkLogic World Amsterdam and meet us there!
2. OWLIM
   In our project at Newz we needed a Big Content Triple Store. We found OWLIM by Ontotext to provide an excellent Big Triple Store, as did BBC and Press Association.
   W3C maintains a list of Big Triple Stores, with BigOWLIM as one of the top products.
   We also selected Ontotext as our partner for their Semantic Tagging capabilities.
3. SOLR
   We have also implemented SOLR for Big Content collections. SOLR will not face all of the Big Content challenges, but is a great Open Source search engine.

PS: After writing this blog, I feel like renaming Meta Data to Meta Content. Probably better if I don’t…