Web archiving and data management

Web archives need to become a seamless part of the experience of using the web; they are the web's corporate memory. This thought encapsulated much of the spirit of a conference that I attended on 21 July on the enduring web, an event organised by JISC, the Digital Preservation Coalition and the UK Web Archiving Consortium.

The conference was a well illustrated overview of the challenges facing the development of a usable and resilient infrastructure for ensuring the perenniality of web content. Quite a task actually, and I was particularly interested in the arguments relating to what material to select in the first place for archiving and preservation. Of course, decisions about what to keep and what to discard have long been everyday stuff for archivists, but when applied to the dynamic, restless and often ephemeral nature of web content, the challenge is particularly acute. Since much web activity is about illustrating work in progress and preserving discourse, to what extent should archiving be documenting the authorial and editing processes?

I was struck by how much such issues resemble those relating to data management. A number of questions facing data archivists should also be familiar to data creators and managers: questions relating to selection, as mentioned above, but also to the curation of material which constantly changes as it is enriched and reformulated; which always stands the risk of being lost forever because it is not properly looked after; and which is not always properly recognised as a scholarly output. Interestingly, the analogy with data management was not made at the meeting, which was attended essentially by librarians and archivists (I had to leave early; perhaps the issue was raised at the end of the day). Could web archivists and data managers learn from each other? Are they actually talking to each other?