Digital Finds

joseph dot reeves at thehumanjourney dot net

Open Source in Archaeology: Science at Work!

Feb 10, 2010 by Joseph Reeves

My last post accused archaeological science practitioners of failing at science, and quite often, archaeological theory too.

Of course, there are people doing proper science and archaeology out there and they are talking about it too. More specifically, we're doing it. My last post was enough to encourage Benjamin to post the slides of a recent paper presented to the Institute of Archaeology:

Ducke 2010: Science Without Software no Longer: Archaeological Data Analysis and the Open Source Paradigm

The slides wonderfully demonstrate the point made by Professor Ince in his Guardian article, with examples from an archaeological perspective. Well worth the read.

We have other presentations available for download on our site:

Good Science Requires Open Source

Feb 08, 2010 by Joseph Reeves

There's a fantastic article on the Guardian's website about further revelations emerging from the UEA "Climategate" affair. The conclusion? Release the source code of applications you use in scientific endeavours. I said something like this at TAG 2008:

The Guardian 5 February 2010: If you're going to do good science, release the computer code too

Computer code is also at the heart of a scientific issue. One of the key features of science is deniability: if you erect a theory and someone produces evidence that it is wrong, then it falls. This is how science works: by openness, by publishing minute details of an experiment, some mathematical equations or a simulation; by doing this you embrace deniability. This does not seem to have happened in climate research. Many researchers have refused to release their computer programs — even though they are still in existence and not subject to commercial agreements. An example is Professor Mann's initial refusal to give up the code that was used to construct the 1999 "hockey stick" model that demonstrated that human-made global warming is a unique artefact of the last few decades. (He did finally release it in 2005.)

So, if you are publishing research articles that use computer programs, if you want to claim that you are engaging in science, the programs are in your possession and you will not release them then I would not regard you as a scientist; I would also regard any papers based on the software as null and void.

I don't know if anyone has ranted at Professor Darrel Ince, but the reaction I got was fantastic; amongst a general shower of scorn, one person flat out called me a liar and another said that such an issue was too theoretical for a meeting of the Theoretical Archaeology Group.

Open Source Software is gaining momentum within archaeological practice, but perhaps not as quickly as it could be. The reason why? Probably because of the nature of archaeological practice rather than the ability of the tools. Many of the practitioners of archaeological computing (in the UK at least, probably also elsewhere) are ivory tower academics interested in producing published volumes. "Archaeological computing", for most, is little more than producing pretty pictures for books. It barely matters for most if you can access the original data, or the black box software that turned it into an image, because there is no element of experimentation or reproducibility required; you've simply produced an image for a book. Other over worked areas of archaeological computing, such as least cost path analysis, view sheds and calorific analysis of landscape exploitation have such little merit from an archaeological perspective that they barely warrant trying to justify by any scientific measures. I tried to this second point a few years back during my MA and, although the only linkable mention of it I can find is on Jo's blog, I think it went down a little better than when I tried to point out the obvious at TAG. Whilst phenomenology critics may enjoy my paper for demonstrating holes in the practice, it also points out that people don't just walk along the easiest paths they can find, nor are they necessarily interested by the fact they can see something impressive on the horizon.

There's plenty of MSc Archaeological Computing types around, but few of them actually do anything scientific. There's no scientific rigour within archaeological practice; no reproducibility, little experimentation, no openness. No scientific hypothesis to stand up and knock down. Archaeological theory is sometimes played with by archaeological computing practitioners, certainly nobody could work in a theoretical vacuum, but it's often implicit, if not shunned. Many people want to make maps within an accepted and uncritical theoretical wasteland that has largely failed to see the emergence of post-processual thinking.

Archaeological computing is, more often than not, behind the curve on scientific practice and archaeological thinking. Professor Ince highlights the need for openness and scientific rigour; let's just hope that people start doing more to work that would actually involve a need for this approach.

Great writeups of Web 2.0 in a crisis

Feb 04, 2010 by Joseph Reeves

A great number of people, myself included, have been doing what we can to help relief work in Haiti. This is now, rather wonderfully, being picked up in the proper press:

NewScientist 27 January 2010: How crowdsourcing is helping in Haiti

The Guardian 4 February 2010: Meet the Wikipedia of the mapping world

See also Mikel Maron's blog entry, this one by Harry Wood and these video's of Haiti mapping by Eric Marsden: Port au Prince, Haiti.

Girona, Somewhere

Jan 28, 2010 by Joseph Reeves

I'm thinking about going to the OpenStreetMap conference this year, named State of the Map. The question remains though, is it in Spain, Italy or Portugal?

It's good to see that a bunch of mapping enthusiasts are so geographically aware. I'm hoping that the next announcement will see Girona moved to somewhere in southern Britain; that'll save a plane journey into Europe.

BBC Archives release episodes of Chronicle

Jan 21, 2010 by Joseph Reeves

Plenty of vintage TV archaeology released via the BBC Archives to help plug some new content:

To coincide with BBC Radio Four's groundbreaking 'A History of the World in 100 objects" the BBC archive collections team have today released 15 episodes of the long-running archaeology programme 'Chronicle' from the archives.

Cold weather auditory Phenomenology?

Jan 21, 2010 by Joseph Reeves

Sounds like ray guns.

Some great cold weather blog posts:

Dispersion of Sound Waves in Ice Sheets

Ice Recordings Updated

…Sounds like Star Wars Blasters

Archaeological implications for encounters with singing lakes abound; who's for some fieldwork?

Stopped Watch

Jan 12, 2010 by Joseph Reeves

My watch stopped yesterday, first time I've ever had that with a digital:

Arduino 7 segment display examples

Dec 26, 2009 by Joseph Reeves

I was lucky enough to receive an Arduino board from Christmas; yesterday I had a quick play with the example code included, but today I tried to cook something up on my own. I've attached the results to this post.

Both simple examples count on a 7 segment display from 0 to 9. The first displays each digit for a second before moving on, the second uses an analogue input (I've tried with a potentiometer and a photocell) to display the digit for a variable amount of time.



I'll be ordering an ethernet shield soon.

Merry Christmas from OA Digital and

Dec 16, 2009 by Joseph Reeves

There are two new releases from OA Digital and that will hopefully interest those from the world of GIS and surveying. Just in time for Christmas, these show that we put our money where our mouths are when it comes to Open Source and Open Data.

gvSIG OADE 2010 Beta

Released today is the beta version of OA Digital's gvSIG 2010 version. Read all about it here:

Despite having the word beta in the title, this is a mature and stable piece of software, although the final version (likely to be released in January) will bring some further polish and improvements. The software is based upon the official gvSIG 1.9 release, but features some heavy tweaking throughout. Checkout the release page for full details and links to the Windows and Linux binaries.

Survey and GIS Manuals

Our Open Data comes in the form of Anna Hodgkinson's Survey and GIS Manuals released through our site. From the project description:

It is meant to supply an easy-to-understand but comprehensive guide to survey and open source GIS, from setting up survey equipment to downloading and processing survey data. The manual is intended for speeding-up, or annihilating the training procedure and providing a guide to survey to field staff in case no professional surveyor is on site. An inexperienced member of field staff should, by following this manual step-by-step, be able to set up a Total Station or a GPS, conduct survey and download and process the survey data, given a certain amount of time.

Today's release sees the addition of GPS manuals and an updating to the TST documentation. An approach of "release early, release often" is being taken, so please make sure to regularly check back for updates an to provide any feedback you may have.

The documents themselves are available through our unglamorous downloads server; I might beautify this at some point in the future.

Download! Be merry!

39% of UK pubs named on OpenStreetMap

Dec 14, 2009 by Joseph Reeves

It's Monday evening, not the time to be thinking about the pub, so I'll just post up some quick and dirty statistics. The conclusion we draw from these? People are recording UK pubs at an encouraging rate although if you see a pub in the wild, chances are it's not on OSM.

Thanks to everyone that sent me messages in the week to tell me that they'd been adding pubs to the database.

The following pulls out and counts all pubs recorded in the UK with names, as extracted from the database by CloudMade (09/12/09):

joseph@joseph-work:~$ grep -ci '<name>Pub:' united_kingdom_Eating_Drinking.gpx

Or about 39% of the UK total. This minor variation counts all pubs, named or not, that are recorded:

joseph@joseph-work:~$ grep -ci '<name>Pub' united_kingdom_Eating_Drinking.gpx

Or about 40% of the UK total; all the way from 12 Bar to Zuri:

joseph@joseph-work:~$ grep -i '<name>Pub' united_kingdom_Eating_Drinking.gpx | sort | head

<name>Pub:12 bar</name>

<name>Pub:13th Note Cafe</name>


<name>Pub:1901 Bar &amp; Bistro</name>

<name>Pub:26 Smithfield</name>

<name>Pub:2nd West</name>



<name>Pub:3 Tuns</name>

<name>Pub:4 Aces</name><

joseph@joseph-work:~$ grep -i '<name>Pub' united_kingdom_Eating_Drinking.gpx | sort | tail



<name>Pub:Zetland Arms</name>

<name>Pub:Zetland Arms</name>

<name>Pub:Zetland Hotel</name>



<name>Pub:Ziggy&apos;s Nightclub</name>




The number of pubs with the word "closed" (incorrectly) present in their name still stands at 33:

joseph@joseph-work:~$ grep -i '<name>Pub' united_kingdom_Eating_Drinking.gpx | grep -ci 'closed'

See my first post for the source of supporting numbers.

Contrary to comments on my last blog post, I haven't been getting these numbers wrong by a factor of three. Rather amusingly, if you run the command suggested by my kind commentator, you end up with exactly the same numbers I posted, suggesting that not only was I correct, but the individual accusing me of doing it wrong hadn't actually tried out their suggestion themselves. I think the problem came because my blog was eating the results as I'd copied and pasted them as HTML. Have resolved that above now.

Pubs: Extending a completeness survey

Dec 04, 2009 by Joseph Reeves

In my last blog post I counted up the number of pubs provided within CloudMade's (25/11/09) OSM export and compared them to the number of pubs we're told that exist within the UK; in short, about 38% of UK pubs are in OpenStreetMap. I got some really great comments on that last entry [1] and was looking forward to the next release from CloudMade to see the rate that UK pubs were (hopefully) growing.

Gregory Williams posted some interesting numbers drawn directly from the database, rather than my CloudMade numbers. He looked for pubs and bars and found 22523 and 350 respectively; I was hoping that this number represented a big increase in recorded pubs, perhaps indicative of the rate at which the UK is being mapped. I was wrong, however, as the release of CloudMade's 02/12/09 dataset revealed:

joseph@joseph-work:~$ grep -ci 'Pub:' united_kingdom_Eating_Drinking.gpx
joseph@joseph-work:~$ grep -i 'Pub:' united_kingdom_Eating_Drinking.gpx | grep -ci 'closed'

(20 742 / 53 466) * 100 = 38.7947481

Whilst the number of pubs recorded in the UK has risen (as with the number of pubs recorded with the word "closed" in their name), the running total still stands at c.38% of the Guardian's figure. It seems then, that Gregory Williams' SQL query was pulling out 2000 pubs from the database that CloudMade doesn't. I think that's quite interesting and possibly something worth looking into. Gregory also notes that 20.7% of UK postboxes are recorded on OSM; I guess we can say that mappers prefer pubs to posting.

This time around I've taken the CloudMade export and plotted the position of OpenStreetMap's record of UK pubs as a simple map:

Click for the large version.

Gary Jones, our GIS officer, gave me a good run-through QGIS 1.3.0; I imported the CloudMade gpx file, pulled out the pubs and plotted them on top of the coastline export. The pubs as a shape file are available here under the same CC-A-SA-2.0 license as the image above. Apologies for the lack of coastline for the Isle of Man in my image, I tried to add it, but it crashed QGIS and seemed to wreck my saved project file, so I left it be. If anyone else creates any other pub maps, please let me know.

There are certainly gaps in the map above! Is that because there's missing pubs or simply areas without pubs? I guess that's the point of a completeness survey! I can say that there are gaps on the image within which I have certainly enjoyed a pint. Presumably the c.62% of UK pubs that don't appear on the map above are located in both areas that are completely blank and also within partially completed regions. The UK is too big an area to conduct meaningful completeness surveys on; we need official pub number and OpenStreetMap counts for a smaller selection of areas. Perhaps county would be an appropriate scale to start with?

Something to mull over on a dark and cold evening...

[1] If you comment on my blog please be aware that long posts may get wrongly marked as spam. This is a massive pain in the backside, but don't worry about it; I will see your comment and mark it as not-spam and it'll appear as it should do. You can safely ignore any warnings you get.

Pubs: An OpenStreetMap completeness survey

Nov 29, 2009 by Joseph Reeves

OpenStreetMap completeness is something that gets spoken about a lot at OA; we already use OSM data in places, but there are some who feel uneasy about the data when it is provided without any indication of its completeness or quality. I think sometimes these questions can be a little unkind; they're rarely asked of Google, for example, and it often falls upon the OpenStreetMap crowd to highlight inaccuracies in this ubiquitous mapping resource. Early last year I noticed some myself.

Despite my moanings, completeness studies are, however, very important. One of the beauties of OpenStreetMap is that you can download all the data yourself, making these studies uniquely possible; the only country you could do similar analysis for based on Google Maps data is Kenya. Luckily, people are looking at completeness of the map, Muki Haklay, for example, has released comparisons of OSM data showing completeness in March 2008 and October 2009.

Bored yesterday, and stuck inside with a cold, I decided to see if I could come up with my own completeness metric. Muki Haklay's example was concerned with roads, I wanted to try something based on PoIs.

The Guardian provides via their excellent DataBlog the number of pubs in the UK: 53,466. From CloudMade we are able to download a gpx file containing a mention of every restaurant, pub & takeaway within the UK recorded on OpenStreetMap. We can very easily pull out the number of pubs and come up with the percentage figure demonstrating the completeness of pub recording in the UK:

joseph@joseph-work:~$ grep -ci 'Pub:' united_kingdom_Eating_Drinking.gpx

(20 620 / 53 466) * 100 = 38.5665657

Let's round that up and say that 38.57% of the pubs in the UK are recorded on OpenStreetMap.

Whilst we can't be sure of the quality of the data provided by the Guardian, we can look a little closer at the OSM data. Some of it, for example, is pretty bad, other pubs have had the word "closed" added to their names, not something you probably need on a map:

joseph@joseph-work:~$ grep -i 'Pub:' united_kingdom_Eating_Drinking.gpx | grep -ci 'closed'

Regardless, these numbers are small so the 38.5% figure still likely rings true enough. Some might argue that it's a small number, but the level of contributors to OpenStreetMap continues to rise in a very positive rate, as such I'm sure that if this was re-run towards the end of 2010 we'd see a big improvement.

If you're reading this and don't know if your local is on OpenStreetMap, why not take a look; once you've got an account you only need to click the edit button to add it.

An Ashmolean Odyssey

Nov 24, 2009 by Joseph Reeves

Last weekend I toured the new Ashmolean Museum with my good friend Peter. Unfortunately I was incredibly hungover, which contributed to a sense of confusion I felt throughout; Pete seems to have captured this quite well in his write-up. It would be wrong to say that my state of mind could be wholly attributed to a previous evening of Staff Benda Bilili and export lager, however, as the physical form of the newly refurbished museum often echoes it's website's ludicrous URL:

Can't it all just be a bit easier?

One of my favourite pieces in the nearly finished museum was an enormous coffee table styled map of the Mediterranean region, stretching from the UK in the north west to Iran and a bit beyond in the south east. Whilst admiring that amazing part of the world where Europe, Asia and Africa meet I remarked on how close the UK was. Drivable in fact, given enough will. Just as I uttered these words and waved my hands about a bit we were approached by an elderly couple who told us they had driven a Ford Escort from London to Islamabad in the 1970s to deliver supplies to the British embassy.

It's a shame that my favourite story in the museum walked out of it sometime that afternoon.

Three inspirational mapping projects

Nov 03, 2009 by Joseph Reeves

Summer in Kabul

AJ Ashton has a really interesting blog post about mashing together Afghani street data, topography and election results for his map Summer in Kabul. AJ writes:

This tile set was created entirely from publicly available data. I used elevation information from the Shuttle Radar Topography Mission, road information from OpenStreetMap, and provincial and district data from Afghanistan Information Management Services. The tools I used were entirely open as well, with the free-software library GDAL handling the elevation interpretations, and Mapnik handling the roads, borders, and labels.


Philippines Ondoy

The Humanitarian OSM Team are calling for people to help map the area affected by Typhoon Ketsana. The group MapAction are already using the results to help victims of the typoon.

DigitalGlobe have donated high quality recent aerial photography for the region; you just need to click here to start tracing the imagery into OpenStreetMap.

Map Kibera

My final inspirational mapping project is probably my favorite of the lot: Map Kibera. The introductory text is simple and powerful:

Kibera, widely known as Africa's largest slum, remains a blank spot on the map. Without basic knowledge of the geography of Kibera it is impossible to have an informed discussion on how to improve the lives of residents of Kibera. Young Kiberans will create the first public digital map of Kibera.

There's a lot more information about the project on the site and it's something I'm going to follow closely; this is exactly the sort of reason I get excited about things like OpenStreetMap (see also my post about Uganda a wee while ago). All three projects are truly inspiring though; Open Data and open tools used to improve people's lives. Fantastic.

Ugandan OpenStreetMap makes the news

Oct 30, 2009 by Joseph Reeves

I've previously written about the import of UN FAO data into OpenStreetMap; now Robert Soden is back, blogging about mapping a Ugandan village.

The news made it to ZDNet with the excellent conclusion:

Now imagine if some proprietary software company or mapping data provider had got involved?

Makes you proud to be an Open Source mapper :)