Category Archives: Infoviz

15 Years of News – Analyzing CNN Transcripts: Visualizing Topics

chart-scatter

High-level visualization of topics in CNN’s corpus

By extracting several topics from our news corpus, we gained a 10,000 feet view of corpus. We were able to outline many trends and events, but it took a bit of digging. This article will illustrate how the topics relate each to one another. We’ll even throw a bit of animation to take a look back the last 15 years.

When we extracted the topics from the corpus, we also found another way to describe all the news snippets and words. Instead of seeing a document as a collection of words, we could see it as a mixture of topics. We could also do the same thing with topics: every topics is a mixture of words used in its associated fragments.

Continue Reading

15 Years of News – Analyzing CNN Transcripts: Topics

High-level time-based visualization of topics in our corpus

High-level visualization of topics in CNN’s corpus

As we saw in the previous article, temporal analysis of individual keywords can be very interesting and uncover interesting trends, but it can be difficult to get an overview of a whole corpus. There’s just too much data.

One solution would be to group similar subjects together. This way, we could scale down a corpus of 500,000 keywords to a handful of topics. Of course, we’ll be losing some definition, but we’ll be gaining a 10,000 feet view of the corpus. In all cases, we can always refer to the individual keywords if we want to take a closer look. Continue Reading

15 Years of News – Analyzing CNN Transcripts: Timelines

Kibana shows that “elections”is a keyword that has a spike every 4 year

With 15 years of CNN transcripts loaded a database, I could now run queries to visualize the occurrences of words – like names – across time. Since I used a textual database name ElasticSearch, I could use Kibana to chart the keywords. Kibana is a good tool to build dashboards, but it’s not really suited to analyze extensively time series because it lacks an easy way to add several search terms on the same chart. Also, it doesn’t easily show percentages of occurrences in a corpus for a giving time period instead of absolute occurrences. This makes Kibana a good tool for a quick look at the data or to debug an issue with our transcript scrapper.

With this in mind, I used Amazon’s DynamoDB database, the HighChart Javascript library and a bit of glue logic to build my own tool to visualize the last 15 years of News!

Continue Reading

Map of scientific collaboration (Redux!)

map_hiSeveral years ago, I created a map of scientific collaborations. The attention this map obtained surpassed my wildest expectations; it got published in the scientific and popular press all around the world! I had mainly forgotten about it until I received an email that rekindled my interest in this visualization and I thought it was high time to revisit this visualization.

Unfortunately, scientific papers (and associated data) are closely guarded and only a handful of firms have full access to them. I now work in a very different field, so I lost access to this dataset. But while perusing my Twitter feed, I came across the very active feed of Scimago Lab. Their social media presence and their incredible interactive visualizations convinced me that they might be interested in collaborating. I sent off an email to their founder, Félix de Moya and, lo and behold, he was interested in collaborating. Cool!

Read on for more maps and an overview of the methodology >>

Continue Reading

Thèmes abordés sur Twitter durant l’élection provinciale de 2012

quebecois_couvSuite à la visualisation des tweets publiés durant la grève étudiante, le professeur Frédérick Bastien de l’Université de Montréal m’a approché afin de participer à l’ouvrage Les Québécois aux urnes.

J’ai donc rédigé un chapitre traitant des thèmes abordés sur les médias sociaux. L’élément central du chapitre était une visualisation de tous les tweets publiés durant la campagne électorale.

Continue Reading

A Map of the Geographical Structure of Wikipedia Links

Wikipedia

Click to enlarge!

There are a lot of Wikipedia visualizations. Some concentrate on article contents, others on the links between articles and some use the geocoded content (like in my previous blog post).

This new visualization is novel because it uses the geographical content of Wikipedia in conjunction with the links between articles. In other words, if a geocoded article (that is, an article associated with a location like a city) links to another geocoded article, a line will be drawn between these two points. The result can be found on the map on the left.

Read on for zoomed views, slideshows, browsable maps, etc.
Continue Reading

A Map of the Geographic Structure of Wikipedia Topics

Wikipedia Topic 260

Mountains, peaks, summits, etc.

A large number of Wikipedia articles are geocoded. This means that when an article pertains to a location, its latitude and longitude are linked to the article. As you can imagine, this can be useful to generate insightful and eye-catching infographics. A while ago, a team at Oxford built this magnificent tool to illustrate the language boundaries in Wikipedia articles. This led me to wonder if it would be possible to extract the different topics in Wikipedia.

This is exactly what I managed to do in the past few days. I downloaded all of Wikipedia, extracted 300 different topics using a powerful clustering algorithm, projected all the geocoded articles on a map and highlighted the different clusters (or topics) in red. The results were much more interesting than I thought. For example, the map on the left shows all the articles related to mountains, peaks, summits, etc. in red on a blue base map.  The highlighted articles from this topic match the main mountain ranges exactly.

Read on for more details, pretty pictures and slideshows.
Continue Reading

Géolocalisation de #qc2012

On a beaucoup parlé des utilisateurs des médias sociaux durant la dernière campagne électorale (#qc2012 pour les intimes). Qui sont-ils? D’où viennent-ils? De quoi parlent-ils?

Les analyses de contenu peuvent répondre à certaines de ces questions, mais pour répondre à la question « où sont-ils? », il faut soit procéder par sondage ou utiliser les données que Twitter nous offre si gentiment. Si les utilisateurs de Twitter activent la fonction de localisation, il est possible de savoir où le “twitteur” est situé. Il suffit ensuite d’importer les données dans un logiciel de visualisation géographique ou Google Tables pour obtenir une carte interactive.

Continue Reading

IndiceQuebec: Perception de la campagne par les médias sociaux

Avec mon collaborateur Eric Nguyen, nous avons créé une petite plateforme d’analyse des tweets politiques Québécois, soit IndiceQuebec.com Cette plateforme intercepte les micromessages contenant certains mots-clés comme #polqc ou #qc2012. Les messages sont ensuite stockés dans une base de données, puis analysés à l’aide de multiples modules. En date d’aujourd’hui, soit presque un mois après les élections, 1,5 millions de tweets sont stockés dans la base de données.

Le module le plus intéressant est celui analysant la teneur du message et détermine si le message est plutôt positif ou négatif. Ce module repose sur un modèle bayésien (même principe qu’un filtre de polluriels) qui détermine à l’aide d’un modèle entraîné au préalable sur un échantillon de tweets. Le modèle est ensuite appliqué sur l’ensemble des tweets au fur à mesure que les micromessages sont emmagasinés dans la base de données.

Continue Reading

#debatQC – Analyse

Le débat des chefs durant la dernière campagne électorale était le premier débat où une quantité significative de personnes ont pu commenter le débat à l’aide des médias sociaux. En utilisant l’engin de récupération des Tweets d’IndiceQuebec, nous avons pu récupérer l’ensemble ou du moins une bonne partie des tweets politiques publiés par les Québécois actifs sur Twitter.

L’image à gauche est une analyse du premier débat à Radio-Canada. La deuxième image (cliquez sur Read more) traite des faces à faces diffusées au réseau TVA.

Continue Reading