Contextual Factors for Finding Similar Experts by Katja Hofmann, Krisztian Balog, Toine Bogers and Maarten de Rijke was accepted by the Journal of the American Society for Information Science and Technology late last year. It is available online now at http://dx.doi.org/10.1002/asi.21292.

Expertise-seeking research studies how people search for expertise and choose whom to contact in the context of a specific task. An important outcome are models that identify factors that influence expert finding. Expertise retrieval addresses the same problem, expert finding, but from a system-centered perspective. The main focus has been on developing content-based algorithms similar to document search. These algorithms identify matching experts primarily on the basis of the textual content of documents with which experts are associated. Other factors, such as the ones identified by expertise-seeking models, are rarely taken into account. In this article, we extend content-based expert-finding approaches with contextual factors that have been found to influence human expert finding. We focus on a task of science communicators in a knowledge-intensive environment, the task of finding similar experts, given an example expert. Our approach combines expertise-seeking and retrieval research. First, we conduct a user study to identify contextual factors that may play a role in the studied task and environment. Then, we design expert retrieval models to capture these factors. We combine these with content-based retrieval models and evaluate them in a retrieval experiment. Our main finding is that while content-based features are the most important, human participants also take contextual factors into account, such as media experience and organizational structure. We develop two principled ways of modeling the identified factors and integrate them with content-based retrieval models. Our experiments show that models combining content-based and contextual factors can significantly outperform existing content-based models.

Listening to ''Stuki'', by The Durutti Column (Play Count: 73)

Our two ECIR 2010 papers are online now. One is entitled Category-based Query Modeling for Entity Search and authored by Krisztian Balog, Marc Bron and Maarten de Rijke. Users often search for entities instead of documents and in this setting are willing to provide extra input, in addition to a query, such as category information and example entities. We propose a general probabilistic framework for entity search to evaluate and provide insight in the many ways of using these types of input for query modeling. We focus on the use of category information and show the advantage of a category-based representation over a term-based representation, and also demonstrate the effectiveness of category-based expansion using example entities. Our best performing model shows very competitive performance on the INEX-XER entity ranking and list completion tasks. The paper is available here.

Our other ECIR 2010 paper is called News Comments: Exploring, Modeling, and Online Prediction with Manos Tsagkias, Wouter Weerkamp and Maarten de Rijke as authors. Online news agents provide commenting facilities for their readers to express their opinions or sentiments with regards to news stories. The number of user supplied comments on a news article may be indicative of its importance, interestingness, or impact. We explore the news comments space, and compare the log-normal and the negative binomial distributions for modeling comments from various news agents. These estimated models can be used to normalize raw comment counts and enable comparison across different news sites. We also examine the feasibility of online prediction of the number of comments, based on the volume observed shortly after publication. We report on solid performance for predicting news comment volume in the long run, after short observation. This prediction can be useful for identifying news stories with the potential to ``take off,'' and can be used to support front page optimization for news sites. The paper is available here.

Listening to ''BWV 0730 Liebster Jesu, wir sind hier'', by Hans Fagius (Organ) (Play Count: 3)

Predicting Podcast Preference: An Analysis Framework and its Application by Manos Tsagkias, Martha Larson and Maarten de Rijke was accepted by Journal of the American Society for Information Science and Technology a while back. It is available online now at http://dx.doi.org/10.1002/asi.21259.

In the paper we start from the observation that finding worthwhile podcasts can be difficult for listeners since podcasts are published in large numbers and vary widely with respect to quality and repute. Independently of their informational content, certain podcasts provide satisfying listening material while other podcasts have little or no appeal. In this paper we present PodCred, a framework for analyzing listener appeal, and we demonstrate its application to the task of automatically predicting the listening preferences of users. First, we describe the PodCred framework, which consists of an inventory of factors contributing to user perceptions of the credibility and quality of podcasts. The framework is designed to support automatic prediction of whether or not a particular podcast will enjoy listener preference. It consists of four categories of indicators related to the Podcast Content, the Podcaster, the Podcast Context, and the Technical Execution of the podcast. Three studies contributed to the development of the PodCred framework: a review of the literature on credibility for other media, a survey of prescriptive guidelines for podcasting, and a detailed data analysis. Next, we report on a validation exercise in which the PodCred framework is applied to a real-world podcast preference prediction task. Our validation focuses on select framework indicators that show promise of being both discriminative and readily accessible. We translate these indicators into a set of easily extractable surface features and use them to implement a basic classification system. The experiments carried out to evaluate system use popularity levels in iTunes as ground truth and demonstrate that simple surface features derived from the PodCred framework are indeed useful for classifying podcasts.

Listening to ''Eleusian Lullaby'', by Alio Die & Martina Galvagni (Play Count: 2)

Conceptual Languages for Domain-Specific Retrieval by Edgar Meij, Dolf Trieschnigg, Maarten de Rijke and Wessel Kraaij was accepted for publication in Information Processing and Management a while back; it is available now. Over the years, various meta-languages have been used to manually enrich documents with conceptual knowledge of some kind. Examples include keyword assignment to citations or, more recently, tags to websites. In this paper we propose generative concept models as an extension to query modeling within the language modeling framework, which leverages these conceptual annotations to improve retrieval. By means of relevance feedback the original query is translated into a conceptual representation, which is subsequently used to update the query model.

Extensive experimental work on five test collections in two domains shows that our approach gives significant improvements in terms of recall, initial precision and mean average precision with respect to a baseline without relevance feedback. On one test collection, it is also able to outperform a text-based pseudo-relevance feedback approach based on relevance models. On the other test collections it performs similarly to relevance models. Overall, conceptual language models have the added advantage of offering query and browsing suggestions in the form of conceptual annotations. In addition, the internal structure of the meta-language can be exploited to add related terms.

Our contributions are threefold. First, an extensive study is conducted on how to effectively translate a textual query into a conceptual representation. Second, we propose a method for updating a textual query model using the concepts in conceptual representation. Finally, we provide an extensive analysis of when and how this conceptual feedback improves retrieval.

iTunes is not playing.

An Efficient Coherence Measure to Determine Topical Consistency in User Generated Content by Jiyin He, Wouter Weerkamp, Martha Larson and Maarten de Rijke is available online. When searching for blogs on a specific topic, information seekers prefer blogs that place a central focus on that topic over blogs whose mention of the topic is diffuse or incidental. In order to present users with better blog feed search results, we develop a measure of topical consistency that is able to capture whether or not a blog is topically focused. The measure, called the coherence score, is inspired by the genetics literature and captures the tightness of the clustering structure of a data set relative to a background collection. In a set of experiments on synthetic data, the coherence score is shown to provide a faithful reflection of topic clustering structure. The properties that make the coherence score more appropriate than lexical cohesion, a common measure of topical structure, are discussed. Retrieval experiments show that integrating the coherence score as a prior in a language modeling-based approach to blog feed search improves retrieval effectiveness. The coherence score must, however, be used judiciously in order to avoid boosting the ranking of irrelevant but topically focused blogs. To this end, we experiment with a series of weighting schemes that adjust the contribution of the coherence score according to the relevance of a blog to the user query. An appropriate weighting scheme is able to improve retrieval performance. Finally, we show that the coherence score can be reliably estimated with a sample exceeding 20 posts in size. Consistent with this finding, experiments show that the best retrieval performance is achieved if coherence scores are used when a blog contains more than 20 posts.

Listening to ''Credo - Chorus Crucifixus'', by Catherine Denley, Etc., Harry Christophers; The Sixteen Choir & Orchestra Catherine Dubosc (Play Count: 5)

Three CIKM 2009 papers are online now. The first, The Impact of Document Structure on Keyphrase Extraction by Katja Hofmann, Manos Tsagkias, Edgar Meij and Maarten de Rijke, can be downloaded here. Keyphrases are short phrases that reflect the main topic of a document. Because manually annotating documents with keyphrases is a time-consuming process, several automatic approaches have been developed. Typically, candidate phrases are extracted using features such as position or frequency in the document text. Document structure may contain useful information about which parts or phrases of a document are important, but has rarely been considered as a source of information for keyphrase extraction. We address this issue in the context of keyphrase extraction from scientific literature. We introduce a new, large corpus that consists of full-text journal articles, where the rich collection and document structure available at the publishing stage is explicitly annotated. We explore features based on the XML tags contained in the documents, and based on generic section types derived using position and cue words in section titles. For XML tags we find sections, abstract, and title to perform best, but many smaller elements may be beneficial in combination with other features. Of the generic section types, the discussion section is found to be the most useful for keyphrase extraction.

The second paper, A Query Model Based on Normalized Log-Likelihood, by Edgar Meij, Wouter Weerkamp and Maarten de Rijke, is available here. Leveraging information from relevance assessments has been proposed as an effective means for improving retrieval. We introduce a novel language modeling method which uses information from each assessed document and their aggregate. While most previous approaches focus either on features of the entire set or on features of the individual relevant documents, our model exploits features of both the documents and the set as a whole. When evaluated, we show that our model is able to significantly improve over state-of-art feedback methods.

The third paper, Predicting the Volume of Comments\\ on Online News Stories by Manos Tsagkias, Wouter Weerkamp and Maarten de Rijke is available here. On-line news agents provide commenting facilities for readers to express their views with regard to news stories. The number of user supplied comments on a news article may be indicative of its importance or impact. We report on exploratory work that predicts the comment volume of news articles prior to publication using five feature sets. We address the prediction task as a two stage classification task: a binary classification identifies articles with the potential to receive comments, and a second binary classification receives the output from the first step to label articles ``low'' or ``high'' comment volume. The results show solid performance for the former task, while performance degrades for the latter.

Learning Semantic Query Suggestions by Edgar Meij, Marc Bron, Laura Hollink, Bouke Huurnink and Maarten de Rijke is available online now. An important application of semantic web technology is recognizing human-defined concepts in text. Query transformation is a strategy often used in search engines to derive queries that are able to return more useful search results than the original query and most popular search engines provide facilities that let users complete, specify, or reformulate their queries. We study the problem of semantic query suggestion, a special type of query transformation based on identifying semantic concepts contained in user queries. We use a feature-based approach in conjunction with supervised machine learning, augmenting term-based features with search history-based and concept-specific features. We apply our method to the task of linking queries from real-world query logs (the transaction logs of the Netherlands Institute for Sound and Vision) to the DBpedia knowledge base. We evaluate the utility of different machine learning algorithms, features, and feature types in identifying semantic concepts using a manually developed test bed and show significant improvements over an already high baseline. The resources developed for this paper, i.e., queries, human assessments, and extracted features, are available for download.

A Generative Blog Post Retrieval Model that Uses Query Expansion based on External Collections by Wouter Weerkamp, Krisztian Balog and Maarten de Rijke is available online now. User generated content is characterized by short, noisy documents, with many spelling errors and unexpected language usage. To bridge the vocabulary gap between the user's information need and documents in a specific user generated content environment, the blogosphere, we apply a form of query expansion, i.e., adding and reweighing query terms. Since the blogosphere is noisy, query expansion on the collection itself is rarely effective but external, edited collections are more suitable. In the paper we propose a generative model for expanding queries using external collections in which dependencies between queries, documents, and expansion documents are explicitly modeled. Different instantiations of our model are discussed and make different (in)dependence assumptions. Results using two external collections (news and Wikipedia) show that external expansion for retrieval of user generated content is effective; besides, conditioning the external collection on the query is very beneficial, and making candidate expansion terms dependent on just the document seems sufficient.

Listening to ''Tears for Affairs'', by Camera Obscura (Play Count: 34)

The University of Amsterdam at WePS2 by Krisztian Balog, Jiyin He, Katja Hofmann, Valentin Jijkoun, Christof Monz, Manos Tsagkias, Wouter Weerkamp and Maarten de Rijke is online now. In this paper we describe our participation in the Second Web People Search workshop (WePS2) and detail our approaches. For the clustering task, our focus was on replicating the lessons learned at WEPS1 on the data set made available as part of WEPS2 and on experimenting with a voting-based combination of clustering methods. We found that clustering methods display the same overall behavior on the WEPS1 and WESP2 data sets and that a hierarchical clustering approach delivers the best performance, even outperforming voting-based combinations.

For attribute extraction, we explore approaches using pattern matching with manually and automatically constructed patterns. Manual patterns were constructed using expert knowledge and following analysis of sample data. Automatic pattern construction extracts textual and syntactic context around training samples and selects patterns which are expected to perform well based on leave-one-out evaluation. Experimental results show that manually constructed patterns are very effective for obtaining high recall. For automatically extracted patterns performance varied widely depending on the attribute type. Larger amounts of training data may help improve these approaches in the future.

Listening to ''Autour de l'arbre'', by Keren Ann (Play Count: 17)

Overview of WebCLEF 2008 by Valentin Jijkoun and Maarten de Rijke is online now. The paper describes the WebCLEF 2008 task. Similarly to the 2007 edition of WebCLEF, the 2008 edition implements a multilingual ``information synthesis" task, where, for a given topic, participating systems have to extract important snippets from web pages. We detail the task, the assessment procedure, the evaluation measures and results.

Listening to ''IV. Allegro'', by J.C. Schickhardt (Play Count: 0)

Concept Models for Domain-specific Search by Edgar Meij and Maarten de Rijke is online now. In the paper we describe our participation in the 2008 CLEF Domain-specific track. We evaluate blind relevance feedback models and concept models on the CLEF domain-specific test collection. Applying relevance modeling techniques is found to have a positive effect on the 2008 topic set, in terms of mean average precision and precision@10. Applying concept models for blind relevance feedback, results in even bigger improvements over a query-likelihood baseline, in terms of mean average precision and early precision.

Listening to ''Struggle For Pleasure'', by Wim Mertens (Play Count: 5)

Using Contextual Information to Improve Search in Email Archives, an ECIR 2009 paper by Wouter Weerkamp, Krisztian Balog, and Maarten de Rijke is available online now. In the paper, we address the task of finding topically relevant email messages in public discussion lists. We make two important observations. First, email messages are not isolated, but are part of a larger online environment. This context, existing on different levels, can be incorporated into the retrieval model. We explore the use of thread, mailing list, and community content levels, by expanding our original query with term from these sources. We find that query models based on contextual information improve retrieval effectiveness. Second, email is a relatively informal genre, and therefore offers scope for incorporating techniques previously shown useful in searching user-generated content. Indeed, our experiments show that using query-independent features (email length, thread size, and text quality), implemented as priors, results in further improvements.

Listening to ''Meu Mundo Hojo (Eu Sou Assim)'', by Teresa Cristina (Play Count: 2)

Two ECIR 2009 papers are online now. The first is Exploiting Surface Features for the Prediction of Podcast Preference by Manos Tsagkias, Martha Larson and Maarten de Rijke. Podcasts display an unevenness characteristic of domains dominated by user generated content, resulting in potentially radical variation of the user preference they enjoy. In the paper we report on work that uses easily extractable surface features of podcasts in order to achieve solid performance on two podcast preference prediction tasks: classification of preferred vs. non-preferred podcasts and ranking podcasts by level of preference. We identify features with good discriminative potential by carrying out manual data analysis, resulting in a refinement of the indicators of an existent podcast preference framework. Our preference prediction is useful for topic-independent ranking of podcasts, and can be used to support download suggestion or collection browsing.

The second paper is Investigating the Global Semantic Impact of Speech Recognition Error on Spoken Content Collections by Martha Larson, Manos Tsagkias, Jiyin He and Maarten de Rijke. Errors in speech recognition transcripts have a negative impact on the effectiveness of content-based speech retrieval and present a particular challenge for collections containing conversational spoken content. We propose a Global Semantic Distortion (GSD) metric that measures the collection-wide impact of speech recognition error on spoken content retrieval in a query-independent manner. We deploy our metric to examine the effects of speech recognition substitution errors. First, we investigate frequent substitutions, cases in which the recognizer habitually mis-transcribes one word as another. Although habitual mistakes have a large global impact, the long tail of rare substitutions has a more damaging effect. Second, we investigate semantically similar substitutions, cases in which the word spoken and the word recognized do not diverge radically in meaning. Similar substitutions are shown to have slightly less global impact than semantically dissimilar substitutions.

See the Publications page.

Listening to ''Polly'', by Keren Ann (Play Count: 15)

A number of new papers have become available online since the last update:

• The University of Amsterdam at TREC 2008: Blog, Enterprise, and Relevance Feedback, K. Balog, E. Meij, W. Weerkamps, J. He, and M. de Rijke. In: TREC 2008 Working Notes, November 2008.
  
  
• The MediaMill TRECVID 2008 Semantic Video Search Engine, C.G.M. Snoek, K.E.A. van de Sande, O. de Rooij, B. Huurnink, J.C. van Gemert, J.R.R. Uijlings, J. He, X. Li, I. Everts, V. Nedovic, M. van Liempt, R. van Balen, F. Yan, M.A. Tahir, K. Mikolajczyk, J. Kittler, M. de Rijke, J.M. Geusebroek, Th. Gevers, M. Worring, A.W.M. Smeulders, and D.C. Koelma. In: TRECvid Working Notes, November 2008.
  
  
• The University of Amsterdam at the TAC 2008 Question Answering Track, V. Jijkoun and M. de Rijke. In: TAC 2008 Working Notes, November 2008.
  
  
• PodCred: A Framework for Analyzing Podcast Preference, M. Tsagkias, M. Larson, W. Weerkamp and M. de Rijke. In: Second Workshop on Information Credibility on the Web (WICOW 2008), October 2008.
  
  
• Non-Local Evidence for Expert Finding, K. Balog and M. de Rijke. In ACM 17th Conference on Information and Knowledge Managment (CIKM 2008), October 2008.
  
  
• Assessing Concept Selection for Video Retrieval, B. Huurnink, K. Hofmann, and M. de Rijke. In: ACM International Conference on Multimedia Information Retrieval (MIR 2008), October 2008.
  
  
• On the Topical Structure of the Relevance Feedback Set, J. He, M. Larson and M. de Rijke. In: FGIR Workshop on Information Retrieval 2008, October 2008.
  
  
• Overview of WebCLEF 2008 (draft), V. Jijkoun and M. de Rijke. In: CLEF 2008 Working Notes, September 2008.
  
  
• A Language Modeling Framework for Expertise Search, K. Balog, L. Azzopardi, and M. de Rijke. Information Processing and Management, doi:10.1016/j.ipm.2008.06.003
  

iTunes is not playing.

The University of Amsterdam at the CLEF 2008 Domain Specific Track by Edgar Meij and Maarten de RIjke is available online now. In the paper we describe our participation in the CLEF 2008 Domain Specific track. The research questions we address are threefold: (i) what are the effects of estimating and applying relevance models to the domain specific collection used at CLEF 2008, (ii) what are the results of parsimonizing these relevance models, and (iii) what are the results of applying concept models for blind relevance feedback? Parsimonization is a technique by which the term probabilities in a language model may be re-estimated based on a comparison with a reference model, making the resulting model more sparse and to the point. Concept models are term distributions over vocabulary terms, based on the language associated with concepts in a thesaurus or ontology and are estimated using the documents which are annotated with concepts. Concept models may be used for blind relevance feedback, by first translating a query to concepts and then back to query terms. We find that applying relevance models helps significantly for the current test collection, in terms of both mean average precision and early precision. Moreover, parsimonizing the relevance models helps mean average precision on title-only queries and early precision on title+narrative queries. Our concept models are able to significantly outperform a baseline query-likelihood run, both in terms of mean average precision and early precision on both title-only and title+narrative queries.

Listening to ''Come Back Margaret'', by Camera Obscura (Play Count: 4)

Non-Local Evidence for Expert Finding by Krisztian Balog and Maarten de Rijke is available online now. The task addressed in this paper, finding experts in an enterprise setting, has gained in importance and interest over the past few years. Commonly, this task is approached as an association finding exercise between people and topics. Existing techniques use either documents (as a whole) or proximity-based techniques to represent candidate experts. Proximity-based techniques have shown clear precision-enhancing benefits. We complement both document and proximity-based approaches to expert finding by importing global evidence of expertise, i.e., evidence obtained using information that is not available in the immediate proximity of a candidate expert's name occurrence or even on the same page on which the name occurs. Examples include candidate priors, query models, as well as other documents a candidate expert is associated with.

Using the CSIRO data set created for the TREC 2007 Enterprise track we identify examples of non-local evidence of expertise. We then propose modified expert retrieval models that are capable of incorporating both local (either document or snippet-based) evidence and non-local evidence of expertise. Results show that our refined models significantly outperform existing state-of-the-art approaches.

iTunes is not playing.

PodCred: A Framework for Analyzing Podcast Preference by Manos Tsagkias, Martha Larson, Wouter Weerkamp and Maarten de Rijke is available online now. The PodCred framework is a framework for assessing the credibility and quality of podcasts published on the internet. It consists of a series of indicators designed to support prediction of listener preference of one podcast over another, given that both carry comparable informational content. The indicators are grouped into four categories pertaining to the Podcast Content, the Podcaster, the Podcast Context or the Technical Execution of the podcast. We adopt the term ``cred'' as a designation encompassing both credibility (comprising trustworthiness and expertise) and qualitative acceptability to listeners. Our podcast analysis framework is inspired by work on credibility in blogs, another medium dominated by user generated content. The PodCred framework is derived from a review of the literature on credibility for other media, a survey of prescriptive standards for podcasting, and a detailed data analysis of award winning podcasts. The paper concludes with a discussion of future work in which the framework will be applied.

iTunes is not playing.

Assessing Concept Selection for Video Retrieval by Bouke Huurning, Katja Hofmann and Maarten de Rijke is available online now. In the paper we explore the use of benchmarks to address the problem of assessing concept selection in video retrieval systems. Two benchmarks are presented, one created by human association of queries to concepts, the other generated from an extensively tagged collection. They are compared in terms of reliability, captured semantics, and retrieval performance. Recommendations are given for using the benchmarks to assess concept selection algorithms; the assessment is demonstrated on two existing algorithms. The benchmarks are released to the research community.

iTunes is not playing.

Using Term Clouds to Represent Segment-Level Semantic Content of Podcasts by Marguerite Fuller, Manos Tsagias, Eamonn Newman, Jana besser, Martha Larson, Gareth Jones and Maarten de Rijke is available online now. Spoken audio, like any time-continuous medium, is notoriously difficult to browse or skim without support of an interface providing semantically annotated jump points to signal the user where to listen in. Creation of time-aligned metadata by human annotators is prohibitively expensive, motivating the investigation of representations of segment-level semantic content based on transcripts generated by automatic speech recognition (ASR). This paper examines the feasibility of using term clouds to provide users with a structured representation of the semantic content of podcast episodes. Podcast episodes are visualized as a series of sub-episode segments, each represented by a term cloud derived from a transcript generated by automatic speech recognition (ASR). Quality of segment-level term clouds is measured quantitatively and their utility is investigated using a small-scale user study based on human labeled segment boundaries. Since the segment-level clouds generated from ASR-transcripts prove useful, we examine an adaptation of text tiling techniques to speech in order to be able to generate segments as part of a completely automated indexing and structuring system for browsing of spoken audio. Results demonstrate that the segments generated are comparable with human selected segment boundaries.

Listening to ''Gathering Dust'', by The Durutti Column (Play Count: 70)

Integrating Contextual Factors into Topic-centric Retrieval Models for Finding Similar Experts by Katja Hofmann, Krisztian Balog, Toine Bogers, and Maarten de Rijke is available online now. Expert finding has been addressed from multiple viewpoints, including expertise seeking and expert retrieval. The focus of expertise seeking has mostly been on descriptive or predictive models, for example to identify what factors affect human decisions on locating and selecting experts. In expert retrieval the focus has been on algorithms similar to document search, which identify topical matches based on the content of documents associated with experts.

We report on a pilot study on an expert finding task in which we explore how contextual factors identified by expertise seeking models can be integrated with topic-centric retrieval algorithms and examine whether they can improve retrieval performance for this task. We focus on the task of \emph{similar expert finding}: given a small number of example experts, find similar experts. Our main finding is that, while topical knowledge is the most important factor, human subjects also consider other factors, such as reliability, up-to-dateness, and organizational structure. We find that integrating these factors into topical retrieval models can significantly improve retrieval performance.

Listening to ''BWV 0826 Partita #2 in c-moll - 5. Rondeaux'', by Pieter-Jan Belder, harpsicord (Play Count: 6)

Blogger, Stick to your Story: Modeling Topical Noise in Blogs with Coherence Measures by Jiyin He, Wouter Weerkamp, Martha Larson, and Maarten de Rijke is available now. Topical noise in blogs arises when bloggers digress from the central topical thrust of their blogs. We introduce a method to explicitly incorporate a model of topical noise into a language modeling approach to the task of blog distillation. Topical noise is integrated into the model using a coherence score, which reflects the tightness of the topical structure of a blog. Tests performed on the TRECBlog06 corpus show that a naive integration of the coherence score as blog prior fails to achieve performance improvements. Instead, we develop a set of more sophisticated models in which the coherence score is weighted by a function of the blog retrieval score. The proposed models help improve effectiveness of our language modeling approach to the blog distillation task.

Listening to ''Run to Yuki'', by Yuji Nomi (Play Count: 3)

Named Entity Normalization in User Generated Content by Valentin Jijkoun, Mahboob Khalid, Maarten Marx and Maarten de Rijke is available online now. Named entity recognition is important for semantically oriented retrieval tasks, such as question answering, entity retrieval, biomedical retrieval, trend detection, and event and entity tracking. In many of these tasks it is important to be able to accurately normalize the recognized entities, i.e., to map surface forms to unambiguous references to real world entities. Within the context of structured databases, this task (known as record linkage and data de-duplication) has been a topic of active research for more than five decades. For edited content, such as news articles, the named entity normalization (NEN) task is one that has recently attracted considerable attention. We consider the task in the challenging context of user generated content (UGC), where it forms a key ingredient of tracking and media-analysis systems.

A baseline NEN system from the literature (that normalizes surface forms to Wikipedia pages) performs considerably worse on UGC than on edited news: accuracy drops from 80% to 65% for a Dutch language data set and from 94% to 77% for English. We identify several sources of errors: entity recognition errors, multiple ways of referring to the same entity and ambiguous references.

To address these issues we propose five improvements to the baseline NEN algorithm, to arrive at a language independent NEN system that achieves overall accuracy scores of 90% on the English data set and 89% on the Dutch data set. We show that each of the improvements contributes to the overall score of our improved NEN algorithm, and conclude with an error analysis on both Dutch and English language UGC. The NEN system is computationally efficient and runs with very modest computational requirements.

iTunes is not playing.

Finding Key Bloggers, One Post At A Time by Wouter Weerkamp, Krisztian Balog and Maarten de Rijke is available online now. User generated content in general, and blogs in particular, form an interesting and relatively little explored domain for mining knowledge. We address the task of blog distillation: to find blogs that are principally devoted to a given topic, as opposed to blogs that merely happen to discuss the topic in passing. Working in the setting of statistical language modeling, we model the task by aggregating a blogger's blog posts to collect evidence of relevance to the topic and persistence of interest in the topic. This approach achieves state-of-the-art performance. On top of this baseline, we extend our model by incorporating a number of blog-specific features, concerning document structure, social structure, and temporal structure. These blog-specific features yield further improvements.

iTunes is not playing.

Measuring Concept Relatedness Using Language Models by Dolf Trieschnigg, Edag Meij, Maarten de Rijke and Wessel Kraaij is available online now. Over the years, the notion of concept relatedness has attracted considerable attention. A variety of approaches, based on ontology structure, information content, association, or context have been proposed to indicate the relatedness of abstract ideas. We propose a method based on the cross entropy reduction between language models of concepts which are estimated based on document-concept assignments. The approach shows improved or competitive results compared to state-of-the-art methods on two test sets in the biomedical domain.

Listening to ''Nolita'', by Keren Ann (Play Count: 19)

A Few Examples Go A Long Way: Constructing Query Models from Elaborate Query Formulations by Krisztian Balog, Wouter Weerkamp and Maarten de Rijke is available online now. In the paper we address a specific enterprise document search scenario, where the information need is expressed in an elaborate manner. In our scenario, information needs are expressed using a short query (of a few keywords) together with examples of key reference pages. Given this setup, we investigate how the examples can be utilized to improve the end-to-end performance on the document retrieval task. Our approach is based on a language modeling framework, where the query model is modified to resemble the example pages. We compare several methods for sampling expansion terms from the example pages to support query-dependent and query-independent query expansion; the latter is motivated by the wish to increase ``aspect recall,'' and attempts to uncover aspects of the information need not captured by the query.

For evaluation purposes we use the CSIRO data set created for the TREC 2007 Enterprise track. The best performance is achieved by query models based on query-independent sampling of expansion terms from the example documents.

Listening to ''A Serious Version'', by King Tubby & The Aggrovators (Play Count: 6)

Term Clouds as Surrogates for User Generated Speech by Manos Tsagias, Martha Larson and Maarten de Rijke is available online. User generated spoken audio remains a challenge for Automatic Speech Recognition (ASR) technology and content-based audio surrogates derived from ASR-transcripts must be error robust. An investigation of the use of term clouds as surrogates for podcasts demonstrates that ASR term clouds closely approximate term clouds derived from human-generated transcripts across a range of cloud sizes. A user study confirms the conclusion that ASR-clouds are viable surrogates for depicting the content of podcasts.

Listening to ''Allegro Blues'', by Dave Brubeck (Play Count: 2)

Parsimonious Concept Modeling by Edgar Meij, Dolf Trieschnigg, Maarten de Rijke, and Wessel Kraaij is available online now. We introduce a parsimonious conceptual query model whose retrieval performance matches that of relevance models, while it is also able to generate high quality navigation suggestions in the form of concepts.

Listening to ''The Paris Match'', by The Style Council (Play Count: 12)

Parsimonious Relevance Models by Edgar Meij, Wouter Weerkamp, Krisztian Balog and Maarten de Rijke is available online. We describe a method for applying parsimonious language models to re-estimate the term probabilities assigned by relevance models. We apply our method to six topic sets from test collections in five different genres. Our parsimonious relevance models (i) improve retrieval effectiveness in terms of MAP on all collections, (ii) significantly outperform their non-parsimonious counterparts on most measures, and (iii) have a precision enhancing effect, unlike other blind relevance feedback methods.

Listening to ''The Paris Match'', by The Style Council (Play Count: 12)

Personal vs Non-Personal Blogs: Initial Classification Experiments by Erik Elgersma and Maarten de Rijke is available online now. In the poster we address the task of separating personal from non-personal blogs, and report on a set of baseline experiments where we compare the performance on a small set of features across a set of five classifiers. We show that with a limited set of features a performance of up to 90\% can be obtained.

Listening to ''Barrio Vejo'', by Ry Cooder (Play Count: 4)

Bloggers as Experts, by Krisztian Balog, Maarten de Rijke and Wouter Weerkamp is available online now. We address the task of (blog) feed distillation: to find blogs that are principally devoted to a given topic. The task may be viewed as an association finding task, between topics and bloggers; it resembles the expert finding task, for which a range of models have been proposed. We adopt two language modeling-based approaches to expert finding, and determine their effectiveness as feed distillation strategies. The two models capture the idea that a human will often search for key blogs by spotting highly relevant posts (the Posting model) or by taking global aspects of the blog into account (the Blogger model). The Blogger model outperforms the Posting model and delivers state-of-the art performance, out-of-the-box.

Listening to ''Barrio Vejo'', by Ry Cooder (Play Count: 4)

Credibility Improves Topical Blog Post Retrieval by Wouter Weerkamps and Maarten de Rijke is available online now. Topical blog post retrieval is the task of ranking blog posts with respect to their relevance for a given topic. To improve topical blog post retrieval we incorporate textual credibility indicators in the retrieval process. We consider two groups of indicators: post level (determined using information about individual blog posts only) and blog level (determined using information from the underlying blogs). We describe how to estimate these indicators and how to integrate them into a retrieval approach based on language models. Experiments on the TREC Blog track test set show that both groups of credibility indicators significantly improve retrieval effectiveness; the best performance is achieved when combining them.

Listening to ''Lullaby 4 Nina'', by The Durutti Column (Play Count: 7)

Looking at Things Differently: Exploring Perspective Recall for Informal Text Retrieval by Wouter Weerkamp and Maarten de Rijke is available now. The paper will be presented at DIR 2008 this April; it reports on ongoing work where we examine the use of query expansion against multiple external corpora so as to uncover multiple perspective on a given topic. Our working assumption is that uncovering multiple perspectives is especially helpful when searching informal text (blogs, discussion forums, comments, etc).

Listening to ''Thin Blue Flame'', by Josh Ritter (Play Count: 0)

The proceedings versions of two CLEF 2007 papers are online now: Overview of WebCLEF 2007, by Valentin Jijkoun and Maarten de Rijke, and Using Centrality to Rank Web Snippets by the same authors. Also available now is Personal Name Resolution of Web People Search by Leif Azzopardi, Krisztian Balog and Maarten de Rijke; this paper will appear in the WWW 2008 workshop on NLP Challenges in the Information Explosion Era (NLPIX 2008).

Listening to ''Fate (Aka For Soph)'', by The Durutti Column (Play Count: 31)

Two contributions to the TREC 2007 proceedings, Query and Document Models for Enterprise Search by Krisztian Balog, Katja Hofmann, Wouter Weerkamp and Maarten de Rijke, and Language Modeling Approaches to Blog Post and Feed Finding by Breyten Ernsting, Wouter Weerkamp and Maarten de Rijke, are online now.

iTunes is not playing.

Associating People and Documents, by Krisztian Balog and Maarten de Rijke is available now. Since the introduction of the Enterprise Track at TREC in 2005, the task of finding experts has generated a lot of interest within the research community. Numerous models have been proposed that rank candidates by their level of expertise with respect to some topic. Common to all approaches is a component that estimates the strength of the association between a document and a person. Forming such associations, then, is a key ingredient in expertise search models. In this paper we introduce and compare a number of methods for building document-people associations. Moreover, we make underlying assumptions explicit, and examine two in detail: (i) independence of candidates, and (ii) frequency is an indication of strength. We show that our refined ways of estimating the strength of associations between people and
documents leads to significant improvements over the state-of-the-art in the end-to-end expert finding task.

Listening to ''Lock Jaw'', by Dave Barker With Tommy Mccook & The Upsetters (Play Count: 1)

Using Coherence-based Measures to Predict Query Difficulty by Jiyin He, Martha Larson and Maarten de Rijke is online now. In the paper we investigate the potential of coherence-based scores to predict query difficulty. The coherence of a document set associated with each query word is used to capture the quality of a query topic aspect. A simple query coherence score, QC-1, is proposed that requires the average coherence contribution of individual query terms to be high. Two further query scores, QC-2 and QC-3, are developed by constraining QC-1 in order to capture the semantic similarity among query topic aspects. All three query coherence scores show the correlation with average precision necessary to make them good predictors of query difficulty. Simple and efficient, the measures require no training data and are competitive with language model-based clarity scores.

Listening to ''Clampdown'', by The Clash (Play Count: 1)

The Impact of Named Entity Normalization on Information Retrieval for Question Answering by Mahboob Alam Khalid, Valentin Jijkoun and Maarten de Rijke is available now. In the named entity normalization task, a system identifies a canonical unambiguous referent for names like Bush or Alabama. Resolving synonymy and ambiguity of such names can benefit end-to-end information access tasks. We evaluate two entity normalization methods based on Wikipedia in the context of both passage and document retrieval for question anwering. We find that even a simple normalization method leads to improvements of early precision, both for document and passage retrieval. Moreover, better normalization results in better retrieval performance.

Listening to ''Everybody Knows This Is Nowhere'', by Neil Young (Play Count: 11)

Extracting the Discussion Structure in Comments on News-Articles by Anne Schuth, Maarten Marx and Maarten de Rijke has now been published. Several on-line daily newspapers offer readers the opportunity to directly comment on articles. In the Netherlands this feature is used quite often and the quality (grammatically and content-wise) is surprisingly high. We develop techniques to collect, store, enrich and analyze these comments. After giving a high-level overview of the Dutch `commentosphere' we zoom in on extracting the discussion structure found in flat comment threads; people not only comment on the news article, they also heavily comment on other comments, resembling discussion fora. We show how techniques from information retrieval, natural language processing and machine learning can be used to extract the `reacts-on' relation between comments with high precision and recall.

Some of the papers describing our participation in TREC this year are available now. For our work on the blog track, see this one, and go here for our work on the enterprise track.

Listening to ''Gathering Dust'', by The Durutti Column (Play Count: 35)