Table of Contents
Comments
Commenters2
6.1 This impact assessment is based on information that was provided by LDAP based on their research on the options for the legal deposit of UK online publications, which are available free of charge and without access restrictions. DCMS have only completed the summary sheets.
6.1 This impact assessment is based on information that was provided by LDAP based on their research on the options for the legal deposit of UK online publications, which are available free of charge and without access restrictions. DCMS have only completed the summary sheets.4
Summary
6.2. LDAP agreed on two possible options for deposit, including both voluntary and regulation-based approaches. In addition, publisher archiving was considered, as an alternative to deposit schemes.
Summary6.2. LDAP agreed on two possible options for deposit, including both voluntary and regulation-based approaches. In addition, publisher archiving was considered, as an alternative to deposit schemes.
5
LDL Permissions-based Harvesting and Archiving (Option One)
6.3. This option might be considered as the equivalent of a voluntary code for deposit, in that it will not be subject to regulation, but it presumes that legal deposit Libraries will actively harvest (‘pull’) material, rather than imposing any duty or encouragement upon publishers to deposit. Thus the main burden of activity rests on the libraries to take periodic copies of websites and documents, then ingest them into an archive for preservation. However libraries may only do so after obtaining copyright permission in writing from the publishers or owners of the material; in responding to these requests, online publishers must give permission on behalf of third party rights-holders.
LDL Permissions-based Harvesting and Archiving (Option One)6.3. This option might be considered as the equivalent of a voluntary code for deposit, in that it will not be subject to regulation, but it presumes that legal deposit Libraries will actively harvest (‘pull’) material, rather than imposing any duty or encouragement upon publishers to deposit. Thus the main burden of activity rests on the libraries to take periodic copies of websites and documents, then ingest them into an archive for preservation. However libraries may only do so after obtaining copyright permission in writing from the publishers or owners of the material; in responding to these requests, online publishers must give permission on behalf of third party rights-holders.
6
LDL Regulation-based Harvesting and Archiving (Option Two)
6.4. This option presumes that libraries, or agents on their behalf, will actively harvest (‘pull’) material, rather than imposing any duty or encouragement upon publishers to deposit. Thus the main burden of activity rests on the Libraries to take periodic copies of freely available online publications and documents, then ingest them into an archive. However regulation would allow libraries to do so without needing to contact the publishers or owners of the material.
LDL Regulation-based Harvesting and Archiving (Option Two)6.4. This option presumes that libraries, or agents on their behalf, will actively harvest (‘pull’) material, rather than imposing any duty or encouragement upon publishers to deposit. Thus the main burden of activity rests on the Libraries to take periodic copies of freely available online publications and documents, then ingest them into an archive. However regulation would allow libraries to do so without needing to contact the publishers or owners of the material.
7
6.5. Regulation will also protect both Libraries and online publishers, to a certain degree, from defamation claims and similar liabilities. However to benefit from these protections, all terms of the 2003 Act and regulations must be applied so that access to material harvested must be restricted to legal deposit library premises. Regulation allows the libraries to harvest more effectively, but will not impose on them any specific requirement, other than a ‘reasonable’ duty to collect.
6.5. Regulation will also protect both Libraries and online publishers, to a certain degree, from defamation claims and similar liabilities. However to benefit from these protections, all terms of the 2003 Act and regulations must be applied so that access to material harvested must be restricted to legal deposit library premises. Regulation allows the libraries to harvest more effectively, but will not impose on them any specific requirement, other than a ‘reasonable’ duty to collect.8
Archiving Left to the Market (Option Three)
6.6. This option proposes there is no central, public sector initiative, and that archiving would be left to the private sector and market forces. Moreover, Libraries would not actively seek to harvest or build a national archive for freely available online publications. Individual publishers might approach a library to archive material on its behalf; this would be subject to separate negotiations and cost-benefit analysis for the specific circumstances.
Archiving Left to the Market (Option Three)6.6. This option proposes there is no central, public sector initiative, and that archiving would be left to the private sector and market forces. Moreover, Libraries would not actively seek to harvest or build a national archive for freely available online publications. Individual publishers might approach a library to archive material on its behalf; this would be subject to separate negotiations and cost-benefit analysis for the specific circumstances.
9
6.7. For each option, the benefits to stakeholders – including publishers, libraries, and members of the public – have been assessed against the overall administrative costs. In addition, other impacts, such as potential risks, were considered. Taken together, these form the evidence base for determining the most appropriate option for deposit.
6.7. For each option, the benefits to stakeholders – including publishers, libraries, and members of the public – have been assessed against the overall administrative costs. In addition, other impacts, such as potential risks, were considered. Taken together, these form the evidence base for determining the most appropriate option for deposit.11
Benefits of a Comprehensive Collection
6.8. The 2003 Act recognises the benefit of collecting and preserving a comprehensive collection of UK electronic publications, including those that are online and freely available. A permanent, comprehensive collection, centrally located and easily accessible to such public stakeholders as researchers, educators, and students enhances UK research and scholarship.
Benefits of a Comprehensive Collection6.8. The 2003 Act recognises the benefit of collecting and preserving a comprehensive collection of UK electronic publications, including those that are online and freely available. A permanent, comprehensive collection, centrally located and easily accessible to such public stakeholders as researchers, educators, and students enhances UK research and scholarship.
14
1. Permission-based Harvesting (Option One)
Permission is only successfully obtained in 30% of cases (based on UKWAC figures). Therefore out of 4500 selected, permission would actually be obtained for only 1350. The archive will therefore represent only a very small proportion (up to approximately 0.5%) of material in scope and perhaps 99.4% may be lost.
Permission is only successfully obtained in 30% of cases (based on UKWAC figures). Therefore out of 4500 selected, permission would actually be obtained for only 1350. The archive will therefore represent only a very small proportion (up to approximately 0.5%) of material in scope and perhaps 99.4% may be lost.
15
2. Regulation-based Harvesting (Option Two)
Regulation ensures a more comprehensive collection, as permissions are not required. It is estimated that up to 79% of material in scope could be captured and archived within available resources.
2. Regulation-based Harvesting (Option Two)Regulation ensures a more comprehensive collection, as permissions are not required. It is estimated that up to 79% of material in scope could be captured and archived within available resources.
16
3. Archiving Left to the Market (Option Three)
Whilst some professional publishers do archive their own material, all the available evidence suggests that the vast majority of individual publishers of freely available online publications within the UK domain do not. Therefore a substantial proportion of the UK’s cultural and intellectual output will be lost forever.
3. Archiving Left to the Market (Option Three)Whilst some professional publishers do archive their own material, all the available evidence suggests that the vast majority of individual publishers of freely available online publications within the UK domain do not. Therefore a substantial proportion of the UK’s cultural and intellectual output will be lost forever.
17
6.10. The collection must be made available to future generations of researchers, or else the benefit will be lost, and costs will accrue to the public as well as Libraries and publishers.
6.10. The collection must be made available to future generations of researchers, or else the benefit will be lost, and costs will accrue to the public as well as Libraries and publishers.18
6.11. Option One not only freely delivers the publications to the public at legal deposit Libraries’ premises, but also has the potential to deliver straight to the users’ PCs. However, although not subject to the premises restrictions required by the 2003 Act, the permissions obtained within Option One might include restrictions to access imposed by individual publishers. Not only might this lead to additional cost to libraries, but it would also affect to what extent researchers and others had access to a comprehensive collection.
6.11. Option One not only freely delivers the publications to the public at legal deposit Libraries’ premises, but also has the potential to deliver straight to the users’ PCs. However, although not subject to the premises restrictions required by the 2003 Act, the permissions obtained within Option One might include restrictions to access imposed by individual publishers. Not only might this lead to additional cost to libraries, but it would also affect to what extent researchers and others had access to a comprehensive collection.19
6.12. Option Two ensures the availability of a permanent and comprehensive collection, as the collecting and archiving of publications is subject to legislation. However, it is a collection subject to the terms of the 2003 Act, restricting the provision of access to within the premises of the Legal Deposit Libraries.
6.12. Option Two ensures the availability of a permanent and comprehensive collection, as the collecting and archiving of publications is subject to legislation. However, it is a collection subject to the terms of the 2003 Act, restricting the provision of access to within the premises of the Legal Deposit Libraries.20
6.13. With Option Three, the uncertainty of the access arrangements and cost to researchers mean that wide availability cannot be guaranteed. In addition, Option Three cannot assure the long-term preservation of an archive, especially when publishers go out of business.
6.13. With Option Three, the uncertainty of the access arrangements and cost to researchers mean that wide availability cannot be guaranteed. In addition, Option Three cannot assure the long-term preservation of an archive, especially when publishers go out of business.21
6.14. Finally, with Options One and Two, the nation would be likely to reap economic benefits from activity associated with international interest in centrally located, comprehensive national collections. Especially in the case of Option Two, where the collection would be more comprehensive, the archive would be of value to overseas researchers interested in the UK. Therefore it is likely to warrant their investment of time and money in the UK for subsidiary activities relating to its use as a resource, whether they are academic or business activities.
6.14. Finally, with Options One and Two, the nation would be likely to reap economic benefits from activity associated with international interest in centrally located, comprehensive national collections. Especially in the case of Option Two, where the collection would be more comprehensive, the archive would be of value to overseas researchers interested in the UK. Therefore it is likely to warrant their investment of time and money in the UK for subsidiary activities relating to its use as a resource, whether they are academic or business activities.22
6.15. Although Options One and Three do not impose any regulatory impact on publishers and Libraries, only Option Two, despite premises restrictions, can fulfil the goal of a deposit scheme, as defined in the 2003 Act: to deliver a comprehensive collection, both freely available and centrally located, to future generations.
6.15. Although Options One and Three do not impose any regulatory impact on publishers and Libraries, only Option Two, despite premises restrictions, can fulfil the goal of a deposit scheme, as defined in the 2003 Act: to deliver a comprehensive collection, both freely available and centrally located, to future generations.23
Question 14: Do you agree with the analysis on making content available to the Deposit Libraries? Explain why. What else needs to be taken into consideration?
Question 14: Do you agree with the analysis on making content available to the Deposit Libraries? Explain why. What else needs to be taken into consideration?24
Financial Implications of Deposit
6.16. These benefits must be balanced against administrative and other costs for the major stakeholders, i.e. Libraries and especially publishers.
Financial Implications of Deposit6.16. These benefits must be balanced against administrative and other costs for the major stakeholders, i.e. Libraries and especially publishers.
25
6.17. The harvesting options for deposit imposes most, if not all, the costs on Libraries. Using the cost model (see Libraries’ Key Costs Assumptions below), which is a realistic view of what could be achieved within the limitations of available resources, costs can be calculated for each of the Options:
6.17. The harvesting options for deposit imposes most, if not all, the costs on Libraries. Using the cost model (see Libraries’ Key Costs Assumptions below), which is a realistic view of what could be achieved within the limitations of available resources, costs can be calculated for each of the Options:26
1. Permissions-based harvesting
Av. total cost per annum £496,959
Material gathered after 10 years (benefit) 77 Terabytes (0.5%)
Cost per terabyte £6,476
1. Permissions-based harvestingAv. total cost per annum £496,959
Material gathered after 10 years (benefit) 77 Terabytes (0.5%)
Cost per terabyte £6,476
27
2. Regulation-based harvesting
Av. total cost per annum £1,132,286
Material gathered after 10 years (benefit) 5,263 Terabytes (79%)
Cost per terabyte £215
2. Regulation-based harvestingAv. total cost per annum £1,132,286
Material gathered after 10 years (benefit) 5,263 Terabytes (79%)
Cost per terabyte £215
28
3. Rely on the market
Av. total cost per annum £ Nil
Material gathered after 10 years (benefit) 0 Terabytes (0%)
Cost per terabyte n/a
3. Rely on the marketAv. total cost per annum £ Nil
Material gathered after 10 years (benefit) 0 Terabytes (0%)
Cost per terabyte n/a
29
6.18. Regulation-based harvesting (Option Two) delivers a larger amount of publications for the lowest unit costs.
6.18. Regulation-based harvesting (Option Two) delivers a larger amount of publications for the lowest unit costs.30
Question 15: Do you agree with this costing model? Explain why. What else needs to be taken into consideration?
Question 15: Do you agree with this costing model? Explain why. What else needs to be taken into consideration?31
Other Costs and Impacts
6.19. In addition to harvesting costs for Libraries, each Option imposes other costs and impacts on stakeholders.
Other Costs and Impacts6.19. In addition to harvesting costs for Libraries, each Option imposes other costs and impacts on stakeholders.
32
6.20. Option One represents a labour-intensive process for both Libraries and publishers to obtain and respond to copyright permission, and this severely constrains the number of websites and volume of material that can be targeted. For Libraries, there is a relatively high staff cost associated with identifying and contacting publishers. For publishers, there are direct administrative and public relations overheads related to responding to requests from libraries, possibly resulting in higher costs to customers.
6.20. Option One represents a labour-intensive process for both Libraries and publishers to obtain and respond to copyright permission, and this severely constrains the number of websites and volume of material that can be targeted. For Libraries, there is a relatively high staff cost associated with identifying and contacting publishers. For publishers, there are direct administrative and public relations overheads related to responding to requests from libraries, possibly resulting in higher costs to customers.33
6.21. As permissions-based harvesting represents only a small proportion of the six million or more potential publications, a greater degree of selectivity is required by Libraries. This adds cost and also has an impact upon the benefits of the archive for future research – any selection decision is based upon the current view of what is important in the present and, with the best of knowledge, in the future.
6.21. As permissions-based harvesting represents only a small proportion of the six million or more potential publications, a greater degree of selectivity is required by Libraries. This adds cost and also has an impact upon the benefits of the archive for future research – any selection decision is based upon the current view of what is important in the present and, with the best of knowledge, in the future.34
6.22. Responding to permissions requests and seeking permissions from third parties also result in high administrative costs for publishers, as well as costs associated with customer and public relations, all of which may translate to increased consumer costs. According to the publishers’ survey conducted by LDAP, publishers generally acknowledged that this would be a costly process, especially if there were no procedure in place or if there were a number of requests to process.
6.22. Responding to permissions requests and seeking permissions from third parties also result in high administrative costs for publishers, as well as costs associated with customer and public relations, all of which may translate to increased consumer costs. According to the publishers’ survey conducted by LDAP, publishers generally acknowledged that this would be a costly process, especially if there were no procedure in place or if there were a number of requests to process.35
6.23. For Option Two, regulation-based harvesting, the process can be much more automated (and tools continue to be developed to facilitate the process) than in permissions-based, but the potentially huge volume of material implies significant extra processing and storage costs. Therefore, Libraries must balance the ideal of collecting all available material in a highly automated fashion against their financial and technical limitations.
6.23. For Option Two, regulation-based harvesting, the process can be much more automated (and tools continue to be developed to facilitate the process) than in permissions-based, but the potentially huge volume of material implies significant extra processing and storage costs. Therefore, Libraries must balance the ideal of collecting all available material in a highly automated fashion against their financial and technical limitations.36
6.24. But regulation will not of itself impose any direct and specific cost to Libraries, nor require any commitment from the public purse. For publishers, costs are marginal, as they do not have to process permissions and do not have to ‘push material’ to libraries.
6.24. But regulation will not of itself impose any direct and specific cost to Libraries, nor require any commitment from the public purse. For publishers, costs are marginal, as they do not have to process permissions and do not have to ‘push material’ to libraries.37
6.25. With Option Three, those publishers who do archive do not typically make their archives available for access by researchers as a matter of course, or may charge for doing so. If they do make them available for a fee, it is to be expected that their own costs will be offset by revenue.
6.25. With Option Three, those publishers who do archive do not typically make their archives available for access by researchers as a matter of course, or may charge for doing so. If they do make them available for a fee, it is to be expected that their own costs will be offset by revenue.38
6.26. However, with this Option, cost burdens spread to include other stakeholder groups, namely business and academic researchers; students; educators; and other members of the public. It imposes a direct cost to the public, not just because it falls short of ensuring a comprehensive collection, but also because archives are spread over a range of publishers with diverse access arrangements. In order to access the range of resources, public stakeholders will incur costs in terms of time, travel and subsistence.
6.26. However, with this Option, cost burdens spread to include other stakeholder groups, namely business and academic researchers; students; educators; and other members of the public. It imposes a direct cost to the public, not just because it falls short of ensuring a comprehensive collection, but also because archives are spread over a range of publishers with diverse access arrangements. In order to access the range of resources, public stakeholders will incur costs in terms of time, travel and subsistence.39
6.27. Option Two places the least amount of financial burden on publishers, and incurs costs that can be managed within existing budgets for Libraries, while ensuring a permanent historical and cultural collection for future generations.
6.27. Option Two places the least amount of financial burden on publishers, and incurs costs that can be managed within existing budgets for Libraries, while ensuring a permanent historical and cultural collection for future generations.40
Question 16: Do you agree with the analysis of the costs and the impacts of each option? Explain why. What else needs to be taken into consideration?
Question 16: Do you agree with the analysis of the costs and the impacts of each option? Explain why. What else needs to be taken into consideration?41
Risks and other impacts
6.28. In addition to gathering evidence on benefits and financial implications for each Option, LDAP also identified specific risks and impacts, concerning civil liabilities, illegal content, third party and dynamic content, as well as potential impact on publisher revenue, all of which also influenced its Recommendation.
Risks and other impacts6.28. In addition to gathering evidence on benefits and financial implications for each Option, LDAP also identified specific risks and impacts, concerning civil liabilities, illegal content, third party and dynamic content, as well as potential impact on publisher revenue, all of which also influenced its Recommendation.
42
6.29. Without regulation, any material harvested by the libraries is not subject to the terms of the 2003 Act. Therefore, Option One does not benefit from any of the protections in the Act, such as those for Libraries and publishers against defamation liability.
6.29. Without regulation, any material harvested by the libraries is not subject to the terms of the 2003 Act. Therefore, Option One does not benefit from any of the protections in the Act, such as those for Libraries and publishers against defamation liability.43
6.30. However, even with regulation-based harvesting, as in Option Two, the results of the Copyright Exceptions Consultation may have an impact on preservation, access and use.
6.30. However, even with regulation-based harvesting, as in Option Two, the results of the Copyright Exceptions Consultation may have an impact on preservation, access and use.44
6.31. Publishers using third-party content are usually required to take it down once licenses have expired. With permissions-based harvesting, Option One, publishers would be expected to obtain permission for this material to be used within an archive. However, these publishers are potentially liable if the libraries have harvested their publications under regulation (Option Two), including this third-party content. A Notice and Take Down policy and procedure would protect both libraries and publishers.
6.31. Publishers using third-party content are usually required to take it down once licenses have expired. With permissions-based harvesting, Option One, publishers would be expected to obtain permission for this material to be used within an archive. However, these publishers are potentially liable if the libraries have harvested their publications under regulation (Option Two), including this third-party content. A Notice and Take Down policy and procedure would protect both libraries and publishers.45
6.32. As the harvesting process is partly automated, the libraries risk inadvertently collecting illegal material. Discussions have already taken place with Internet Watch Foundation (IWF) to identify the levels of risk involved and ways in which they might be minimised. IWF has also given advice on the procedures which should be followed if any such material should be found within the archive. Separate discussions have also been held with the Information Commissioner’s Office to address any concerns the Commissioner might have in relation to personal data within the archive. Overall the ICO sees no reason to restrict harvesting activity within the context of legal deposit, nor to require additional controls over access than are implied by the 2003 Act, save for the addendum to the Notice and Take down proposals.
6.32. As the harvesting process is partly automated, the libraries risk inadvertently collecting illegal material. Discussions have already taken place with Internet Watch Foundation (IWF) to identify the levels of risk involved and ways in which they might be minimised. IWF has also given advice on the procedures which should be followed if any such material should be found within the archive. Separate discussions have also been held with the Information Commissioner’s Office to address any concerns the Commissioner might have in relation to personal data within the archive. Overall the ICO sees no reason to restrict harvesting activity within the context of legal deposit, nor to require additional controls over access than are implied by the 2003 Act, save for the addendum to the Notice and Take down proposals.46
6.33. On the subject of permissions-based harvesting, one of the publishers surveyed observed, “Permissions are not an issue. Knowing what information to harvest is the issue. …because our website content changes regularly, such questions would need to be addressed every time a harvest was made.” Publishers are concerned over securing ongoing rights for data or images for sites that were made available free of charge but on a time-limited basis. In the face of dynamic content, Option One does not include the degree of flexibility that is required to gather a large number of publications efficiently, and without placing undue financial burden on publishers and libraries alike.
6.33. On the subject of permissions-based harvesting, one of the publishers surveyed observed, “Permissions are not an issue. Knowing what information to harvest is the issue. …because our website content changes regularly, such questions would need to be addressed every time a harvest was made.” Publishers are concerned over securing ongoing rights for data or images for sites that were made available free of charge but on a time-limited basis. In the face of dynamic content, Option One does not include the degree of flexibility that is required to gather a large number of publications efficiently, and without placing undue financial burden on publishers and libraries alike.47
6.34. Publishers, in the survey conducted at the beginning of 2008, expressed concern about the possible impact on their revenues of deposit of (and immediate access to) time-sensitive and other types of revenue-generating publications. As with print deposit, deposit of freely available online publications can include provisions for special arrangements to offset any potential problem (see paragraphs 6.43 and 6.44 on embargoes). Moreover, risks can be minimised by the restrictions to access imposed by the 2003 Act.
6.34. Publishers, in the survey conducted at the beginning of 2008, expressed concern about the possible impact on their revenues of deposit of (and immediate access to) time-sensitive and other types of revenue-generating publications. As with print deposit, deposit of freely available online publications can include provisions for special arrangements to offset any potential problem (see paragraphs 6.43 and 6.44 on embargoes). Moreover, risks can be minimised by the restrictions to access imposed by the 2003 Act.48
Question 17: Do you agree with risks identified here? Explain why. Are there other risks that have not been considered? What would their impact be? Are some of these risks actually not really risks? Why?
Question 17: Do you agree with risks identified here? Explain why. Are there other risks that have not been considered? What would their impact be? Are some of these risks actually not really risks? Why?50
6.35. Both Options One (permissions-based harvesting) and Three (archiving left to the market) fail to meet the basic requirement of creating a permanent national collection, freely available at designated sites, as envisaged in the 2003 Act.
6.35. Both Options One (permissions-based harvesting) and Three (archiving left to the market) fail to meet the basic requirement of creating a permanent national collection, freely available at designated sites, as envisaged in the 2003 Act.51
6.36. Only Option Two, regulation-based harvesting and archiving, would meet this requirement for the following reasons:
6.36. Only Option Two, regulation-based harvesting and archiving, would meet this requirement for the following reasons:52
• Rate of Deposit: The large number of freely available online publications and the rate of increase (about 15% per annum) expected pose a challenge to creating a comprehensive collection in the most efficient way possible. Harvesting provides the most efficient and timely solution for deposit of publications in this category: regulations-based harvesting specifically ensures the greatest amount can be collected to build the most comprehensive collection for future generations.
• Rate of Deposit: The large number of freely available online publications and the rate of increase (about 15% per annum) expected pose a challenge to creating a comprehensive collection in the most efficient way possible. Harvesting provides the most efficient and timely solution for deposit of publications in this category: regulations-based harvesting specifically ensures the greatest amount can be collected to build the most comprehensive collection for future generations.
53
• Proportionality: LDAP wants to ensure that the costs and scale of publications are proportionate to a Recommendation for regulation. Regulation is an appropriate response to the enormous number of publishers and publications in this category. It is the most cost-efficient method of collection for the libraries and imposes no direct financial or administrative burden upon the publishers.
• Proportionality: LDAP wants to ensure that the costs and scale of publications are proportionate to a Recommendation for regulation. Regulation is an appropriate response to the enormous number of publishers and publications in this category. It is the most cost-efficient method of collection for the libraries and imposes no direct financial or administrative burden upon the publishers.
54
• Existing Harvesting Activities and Infrastructure: Whilst libraries have not yet been able to harvest widely across the whole UK domain, much of the technical infrastructure and expertise that they would need is already in place. They have practical experience of permissions-based harvesting for the UK Web Archiving Consortium. In addition, the International Internet Preservation Consortium (IIPC), of which the BL and NLS are members, is developing tools, infrastructure and the policies for addressing the challenges of collection and preservation internationally. Moreover, knowledge gained from other national libraries that already carry out domain-level harvesting (e.g. in France, Denmark, New Zealand and Australia) will also assist its implementation in the UK. Indeed, the UK legal deposit libraries estimate that they could begin harvesting on this basis, up to 80% in scale, within existing resources and within a very short time of implementation.
• Existing Harvesting Activities and Infrastructure: Whilst libraries have not yet been able to harvest widely across the whole UK domain, much of the technical infrastructure and expertise that they would need is already in place. They have practical experience of permissions-based harvesting for the UK Web Archiving Consortium. In addition, the International Internet Preservation Consortium (IIPC), of which the BL and NLS are members, is developing tools, infrastructure and the policies for addressing the challenges of collection and preservation internationally. Moreover, knowledge gained from other national libraries that already carry out domain-level harvesting (e.g. in France, Denmark, New Zealand and Australia) will also assist its implementation in the UK. Indeed, the UK legal deposit libraries estimate that they could begin harvesting on this basis, up to 80% in scale, within existing resources and within a very short time of implementation.
55
• Civil Liabilities: Regulation would protect both publishers and libraries from such civil liabilities as copyright infringement and defamation. However, there are potential risks that may have significant impacts, if not eventual costs.
• Civil Liabilities: Regulation would protect both publishers and libraries from such civil liabilities as copyright infringement and defamation. However, there are potential risks that may have significant impacts, if not eventual costs.
56
Question 18: Do you agree with LDAP’s recommendation to regulate for this content? If not, what should be done instead?
Question 18: Do you agree with LDAP’s recommendation to regulate for this content? If not, what should be done instead?57
Policies for Deposit, Access and Use
Restrictions on activities in relation to freely available online publications
Policies for Deposit, Access and UseRestrictions on activities in relation to freely available online publications
58
6.37. LDAP recommends the following policies regarding provision for access and use as required by Section 7 of the 2003 Act:
6.37. LDAP recommends the following policies regarding provision for access and use as required by Section 7 of the 2003 Act:59
• No limitations are imposed on access in the specific areas of time and circumstance of use (7.4 (b)) and number of readers (7.4 (d)). LDAP recommends that the standard terms of access already implied by the 2003 Act are correct and that no changes are necessary. Thus access must be confined to readers (and staff) using terminals, screens or devices that are controlled by the Libraries, and whilst they are on the Libraries’ premises.
• No limitations are imposed on access in the specific areas of time and circumstance of use (7.4 (b)) and number of readers (7.4 (d)). LDAP recommends that the standard terms of access already implied by the 2003 Act are correct and that no changes are necessary. Thus access must be confined to readers (and staff) using terminals, screens or devices that are controlled by the Libraries, and whilst they are on the Libraries’ premises.
60
• LDAP does not seek to extend the conditions of access in the specific areas of time and circumstance of use (7.4 (b)) and number of readers (7.4 (d)).
• LDAP does not seek to extend the conditions of access in the specific areas of time and circumstance of use (7.4 (b)) and number of readers (7.4 (d)).
61
• Libraries must not be permitted to adapt or change the intellectual content of the material itself. The original ‘look and feel’ must also be preserved as far as possible (subject to technical limitations).
• Libraries must not be permitted to adapt or change the intellectual content of the material itself. The original ‘look and feel’ must also be preserved as far as possible (subject to technical limitations).
62
• In the case of 7.2 (b), the regulation must allow for a single copy harvested by one Library to be made available in all six Libraries and therefore to be regarded as being the equivalent of depositing six copies (i.e. one per LDL) for storage, preservation and access purposes.
• In the case of 7.2 (b), the regulation must allow for a single copy harvested by one Library to be made available in all six Libraries and therefore to be regarded as being the equivalent of depositing six copies (i.e. one per LDL) for storage, preservation and access purposes.
63
• For preservation purposes, legal deposit Libraries must be allowed to transfer publications from one technology platform to another in the event of upgrades to technology or where changes of systems are required.
• For preservation purposes, legal deposit Libraries must be allowed to transfer publications from one technology platform to another in the event of upgrades to technology or where changes of systems are required.
64
• Libraries must not be permitted to lend or transfer the material, except in the case of other legal deposit libraries. ¥ Libraries must be permitted to copy material for the purposes of preserving the material and ensuring that access can continue to be provided, but not in order to increase access.
• Libraries must not be permitted to lend or transfer the material, except in the case of other legal deposit libraries. ¥ Libraries must be permitted to copy material for the purposes of preserving the material and ensuring that access can continue to be provided, but not in order to increase access.
67
• LDAP requests that there be a note on ‘Removal of Access’ (or ‘notice and take-down’) such that, for material in their collections that is subject to a court injunction or judgement, Libraries, subject to notice, will comply with the courts by denying all access to all readers for an appropriate period, which might perhaps be for many years (this does not include deletion or disposal of a web page).
• LDAP requests that there be a note on ‘Removal of Access’ (or ‘notice and take-down’) such that, for material in their collections that is subject to a court injunction or judgement, Libraries, subject to notice, will comply with the courts by denying all access to all readers for an appropriate period, which might perhaps be for many years (this does not include deletion or disposal of a web page).
68
• In the case of 7.4 (f), Libraries must be permitted to dispose of materials prior to incorporation into the collection, defined as the point before which readers have access.
• In the case of 7.4 (f), Libraries must be permitted to dispose of materials prior to incorporation into the collection, defined as the point before which readers have access.
69
• Moreover, Libraries must not be permitted to dispose of materials once access to readers has been provided.
• Moreover, Libraries must not be permitted to dispose of materials once access to readers has been provided.
71
• Legal deposit Libraries, while not required to grant embargoes, may consider requests based on demonstrable financial prejudice to the requestor, with reference to the specific conditions of access.
• Legal deposit Libraries, while not required to grant embargoes, may consider requests based on demonstrable financial prejudice to the requestor, with reference to the specific conditions of access.
72
6.39. Regarding (7.4 (c)), the Panel would like the Act’s definition of ‘reader’ and ‘relevant person’ to be included where appropriate. (See 2003 Act Section 5)
6.39. Regarding (7.4 (c)), the Panel would like the Act’s definition of ‘reader’ and ‘relevant person’ to be included where appropriate. (See 2003 Act Section 5)73
a) ”reader” means a person who, for the purposes of research or study and with the permission of a deposit library, is on library premises controlled by it;
a) ”reader” means a person who, for the purposes of research or study and with the permission of a deposit library, is on library premises controlled by it;76
Policy Background
6.40. For discussions on Section 7, LDAP was guided by what is permissible in the 2003 Act as well as in print deposit practice as initial points of reference for policy development. These considerations resulted in policy allowing the reader to print, but not download or make electronic copies (‘digital copying’). Libraries will be allowed to make electronic copies for preservation and continued access, but not for lending and transferring. Further deliberation on the subject of printing included whether there should be any limits to the amount or proportion of material that a reader may print out on paper. LDAP acknowledged that this issue might need further work; while some LDAP members felt that no restriction is necessary for this category of material, as it is originally made freely available by the publisher, others felt that a lack of any restriction may have wider ramifications. However, all LDAP members agreed that, in keeping with print deposit practice and in acknowledgement of the nature of the material, a reader should be permitted to make printed copies.
Policy Background6.40. For discussions on Section 7, LDAP was guided by what is permissible in the 2003 Act as well as in print deposit practice as initial points of reference for policy development. These considerations resulted in policy allowing the reader to print, but not download or make electronic copies (‘digital copying’). Libraries will be allowed to make electronic copies for preservation and continued access, but not for lending and transferring. Further deliberation on the subject of printing included whether there should be any limits to the amount or proportion of material that a reader may print out on paper. LDAP acknowledged that this issue might need further work; while some LDAP members felt that no restriction is necessary for this category of material, as it is originally made freely available by the publisher, others felt that a lack of any restriction may have wider ramifications. However, all LDAP members agreed that, in keeping with print deposit practice and in acknowledgement of the nature of the material, a reader should be permitted to make printed copies.
77
6.41. In the matter of disposal (f), while the 2003 Act does not prohibit disposal for print publications, it does contain an express prohibition on disposal for non-print publications. Therefore, LDAP recognises that it is necessary to specify the right to do so in the secondary legislation, for instance of collection of duplicate and partial freely-available online publications. LDAP feels that it is possible to articulate a limited provision of such rights to dispose, perhaps in a guidance note accompanying regulation as it is arguable that duplicate and partial delivery of documents is not effective ‘delivery of relevant material’ under the 2003 Act and cannot be perceived as contravening the requirement not to dispose, as in the first case one copy has already been collected and will be retained, and in the second instance a representative or complete copy has not been effectively collected and could not therefore be considered as ‘disposed’.
6.41. In the matter of disposal (f), while the 2003 Act does not prohibit disposal for print publications, it does contain an express prohibition on disposal for non-print publications. Therefore, LDAP recognises that it is necessary to specify the right to do so in the secondary legislation, for instance of collection of duplicate and partial freely-available online publications. LDAP feels that it is possible to articulate a limited provision of such rights to dispose, perhaps in a guidance note accompanying regulation as it is arguable that duplicate and partial delivery of documents is not effective ‘delivery of relevant material’ under the 2003 Act and cannot be perceived as contravening the requirement not to dispose, as in the first case one copy has already been collected and will be retained, and in the second instance a representative or complete copy has not been effectively collected and could not therefore be considered as ‘disposed’.78
6.42. LDAP sought to distinguish between harvested publications already incorporated into the permanent collection and those yet to be incorporated. Generally speaking, the restrictions against disposal would cover only those publications part of the permanent collection, and that duplicate or incomplete copies would be disposed of before becoming part of the permanent collection.
6.42. LDAP sought to distinguish between harvested publications already incorporated into the permanent collection and those yet to be incorporated. Generally speaking, the restrictions against disposal would cover only those publications part of the permanent collection, and that duplicate or incomplete copies would be disposed of before becoming part of the permanent collection.79
6.43. With regard to 7.4 (b), LDAP’s deliberations on the subject of embargoes originally began with defining and analysing the nature of the publications freely available, free of charge and without access restriction, the collecting or harvesting of these publications, and costs that would be incurred by publishers as a result of such activity. In considering cost, LDAP looked to the BRE guidelines in order to have a better understanding of what costs should be included. Taking into consideration the types of costs highlighted by the BRE and the relative passive role of publishers in the harvesting activity (see description of harvesting above), LDAP’s initial assumption was that as no real (admin) cost was incurred by publishers, and because the material was already freely available on the web, there was no requirement for embargoes. However, in response to subsequent concerns voiced by LDAP members and other stakeholders, LDAP deliberated on two measures that would provide comfort to publisher concerns:
6.43. With regard to 7.4 (b), LDAP’s deliberations on the subject of embargoes originally began with defining and analysing the nature of the publications freely available, free of charge and without access restriction, the collecting or harvesting of these publications, and costs that would be incurred by publishers as a result of such activity. In considering cost, LDAP looked to the BRE guidelines in order to have a better understanding of what costs should be included. Taking into consideration the types of costs highlighted by the BRE and the relative passive role of publishers in the harvesting activity (see description of harvesting above), LDAP’s initial assumption was that as no real (admin) cost was incurred by publishers, and because the material was already freely available on the web, there was no requirement for embargoes. However, in response to subsequent concerns voiced by LDAP members and other stakeholders, LDAP deliberated on two measures that would provide comfort to publisher concerns:80
• Embargoes, the well-established practice for printed publications, in which a publisher of material that is, for example, of high commercial value but sold in small numbers, may contact the Libraries and negotiate a temporary embargo upon access to deposited items for a period, typically 1-2 years. These embargoes and their conditions would be reviewed on a case by case basis. They do not prohibit the collection of material nor eventual access to the material. As a result of considering the print precedent, LDAP was persuaded that embargoes were a reasonable policy to apply in the case of the harvesting of freely-available online publications.
• Embargoes, the well-established practice for printed publications, in which a publisher of material that is, for example, of high commercial value but sold in small numbers, may contact the Libraries and negotiate a temporary embargo upon access to deposited items for a period, typically 1-2 years. These embargoes and their conditions would be reviewed on a case by case basis. They do not prohibit the collection of material nor eventual access to the material. As a result of considering the print precedent, LDAP was persuaded that embargoes were a reasonable policy to apply in the case of the harvesting of freely-available online publications.
81
• Opt-outs, the practice of allowing publishers of certain types of materials to be exempted from the collection and preservation of their freely available publications, and for this exemption to be considered as part of the policy recommendations made by LDAP. Upon further consideration, LDAP felt that opt-outs were not a reasonable policy to apply in the case of the harvesting of freely-available online publications.
• Opt-outs, the practice of allowing publishers of certain types of materials to be exempted from the collection and preservation of their freely available publications, and for this exemption to be considered as part of the policy recommendations made by LDAP. Upon further consideration, LDAP felt that opt-outs were not a reasonable policy to apply in the case of the harvesting of freely-available online publications.
82
6.44. LDAP has considered both options in detail, guided by current practice, research on business models and revenue streams, as well as discussions with publishing stakeholders. While none of these deliberations has produced any causal link between the practice of legal deposit of freely available online publications and the challenge to revenue streams, LDAP felt strongly that publishers’ real concerns for future impact on revenue streams should be accommodated. In the specific case of freely available online publications, legal deposit process provides for a much more controlled environment for access than readers’ own homes (where this material would be accessible in the first instance). The resulting recommendation is a policy for individual publishers to request embargoes on a case by case basis, as an acknowledgement of these concerns, balanced with the mandate of the collection and preservation of these materials set out by the 2003 Act.
6.44. LDAP has considered both options in detail, guided by current practice, research on business models and revenue streams, as well as discussions with publishing stakeholders. While none of these deliberations has produced any causal link between the practice of legal deposit of freely available online publications and the challenge to revenue streams, LDAP felt strongly that publishers’ real concerns for future impact on revenue streams should be accommodated. In the specific case of freely available online publications, legal deposit process provides for a much more controlled environment for access than readers’ own homes (where this material would be accessible in the first instance). The resulting recommendation is a policy for individual publishers to request embargoes on a case by case basis, as an acknowledgement of these concerns, balanced with the mandate of the collection and preservation of these materials set out by the 2003 Act.83
6.45. In addition, it is felt that optional embargoes do not compromise the integrity of a regulation (as does allowing certain publication types to opt-out) , nor the “future-proofing” of that regulation (i.e., the in-built flexibility within a regulation to acknowledge the changing nature of publications as a result of technology and business models dictated by it). This concern for future-proofing has been the basis of the LDAP’s considerations, beginning with the EPS consultation.
6.45. In addition, it is felt that optional embargoes do not compromise the integrity of a regulation (as does allowing certain publication types to opt-out) , nor the “future-proofing” of that regulation (i.e., the in-built flexibility within a regulation to acknowledge the changing nature of publications as a result of technology and business models dictated by it). This concern for future-proofing has been the basis of the LDAP’s considerations, beginning with the EPS consultation.84
6.46. With regard to material currently held by the Libraries which has been deemed potentially prejudicial and therefore inappropriate for public access by the courts, the libraries would presently not be notified of any resulting injunction or judicial direction against the publisher. LDAP understands that the Libraries are aware of risks and current advice on this issue. With regard to inadvertent harvesting of online publications, the publication and/or dissemination of which may be an offence (e.g. under the Terrorism Act 2006 and the Sexual Offences Act 2003), a case may be made to grant libraries an exemption from criminal liability (pursuant to the power to do so under the 2003 Act), for both the act of harvesting and retention of the publication if the Panel and its library members see such immunity as a desirable objective.
6.46. With regard to material currently held by the Libraries which has been deemed potentially prejudicial and therefore inappropriate for public access by the courts, the libraries would presently not be notified of any resulting injunction or judicial direction against the publisher. LDAP understands that the Libraries are aware of risks and current advice on this issue. With regard to inadvertent harvesting of online publications, the publication and/or dissemination of which may be an offence (e.g. under the Terrorism Act 2006 and the Sexual Offences Act 2003), a case may be made to grant libraries an exemption from criminal liability (pursuant to the power to do so under the 2003 Act), for both the act of harvesting and retention of the publication if the Panel and its library members see such immunity as a desirable objective.85
Question 19: Do you agree with LDAP’s proposed method for depositing of content? If not how else could this be done?
Question 19: Do you agree with LDAP’s proposed method for depositing of content? If not how else could this be done?86
Question 20: Do you agree with LDAP’s analysis of access provisions? Explain why. What other options are there? Libraries’ key cost assumptions
Question 20: Do you agree with LDAP’s analysis of access provisions? Explain why. What other options are there? Libraries’ key cost assumptions87
6.47. The category definition and territoriality rules govern what may be collected. Within these, this model assumes that the UK web space is defined as all .uk domains registered by Nominet (6.1 million in mid-2007) plus approximately 50,000 other domains which can be readily identified as published in the UK. The numbers will continue to grow by 17% per annum until 2011, then by 15% till 2016[1].
6.47. The category definition and territoriality rules govern what may be collected. Within these, this model assumes that the UK web space is defined as all .uk domains registered by Nominet (6.1 million in mid-2007) plus approximately 50,000 other domains which can be readily identified as published in the UK. The numbers will continue to grow by 17% per annum until 2011, then by 15% till 2016[1].88
6.48. However 35% of domains are inactive[2], i.e. registered but not live, or where static content can be ‘deduplicated’ after a first snapshot. A further 2.5%[3] are primarily ‘deep web’ or protected sites outside the scope of this category. Overall, the number of websites in scope is therefore estimated as 3.9 million in 2007 rising to 14.6 million in 2016.
6.48. However 35% of domains are inactive[2], i.e. registered but not live, or where static content can be ‘deduplicated’ after a first snapshot. A further 2.5%[3] are primarily ‘deep web’ or protected sites outside the scope of this category. Overall, the number of websites in scope is therefore estimated as 3.9 million in 2007 rising to 14.6 million in 2016.89
6.49. The average size of each site has been growing significantly each year. However this model assumes that most audio-visual content, one of the major causes of growth, is out of scope and therefore a more modest 5% growth per annum is appropriate. The average size also varies dramatically, from circa 5 megabytes for 80% of sites to 1 gigabyte for 0.5% of sites; this model assumes a weighted average of 25 megabytes. However, in permissions-based harvesting selection tends to favour the larger websites, so this model assumes an average 180 megabytes in that option.[4]
6.49. The average size of each site has been growing significantly each year. However this model assumes that most audio-visual content, one of the major causes of growth, is out of scope and therefore a more modest 5% growth per annum is appropriate. The average size also varies dramatically, from circa 5 megabytes for 80% of sites to 1 gigabyte for 0.5% of sites; this model assumes a weighted average of 25 megabytes. However, in permissions-based harvesting selection tends to favour the larger websites, so this model assumes an average 180 megabytes in that option.[4]90
6.50. The cost of storage includes built-in redundancy to ensure safe preservation of the archive; content is replicated (with security measures) across three nodes in the BL/NLW shared infrastructure and across 2 nodes in the NLS infrastructure. However the real cost of storage per terabyte has fallen by more than 30% per annum over the last 20 years and is expected to continue falling by 25% per annum until 2016.[5]
6.50. The cost of storage includes built-in redundancy to ensure safe preservation of the archive; content is replicated (with security measures) across three nodes in the BL/NLW shared infrastructure and across 2 nodes in the NLS infrastructure. However the real cost of storage per terabyte has fallen by more than 30% per annum over the last 20 years and is expected to continue falling by 25% per annum until 2016.[5]91
6.51. The two infrastructures have already being designed and built – apart from certain elements – by the libraries and will be used to store other digital or digitised collections as well as legal deposit material. Therefore this model focuses only on the incremental systems costs – including renewing equipment every three years – salaries, pensions, NI, other staff related costs, plus allocations for general support and overheads, that are required to collect and preserve ‘free web’ material.
6.51. The two infrastructures have already being designed and built – apart from certain elements – by the libraries and will be used to store other digital or digitised collections as well as legal deposit material. Therefore this model focuses only on the incremental systems costs – including renewing equipment every three years – salaries, pensions, NI, other staff related costs, plus allocations for general support and overheads, that are required to collect and preserve ‘free web’ material.92
Question 21: Do you agree with these cost assumptions? Explain why. What needs to be included or excluded?
Question 21: Do you agree with these cost assumptions? Explain why. What needs to be included or excluded?94
[1] Assumption based upon historic UK growth data from Nominet, historic .com/.org/.net data from http://www.zooknic.com and http://www.oecd.org/dataoecd/56/34/32996948.pdf
[1] Assumption based upon historic UK growth data from Nominet, historic .com/.org/.net data from http://www.zooknic.com and http://www.oecd.org/dataoecd/56/34/32996948.pdf97
[4] Assumptions and estimates based upon historic data from UKWAC and national libraries in Denmark, Australia and New Zealand
[4] Assumptions and estimates based upon historic data from UKWAC and national libraries in Denmark, Australia and New Zealand98
[5] Conservative assumptions based upon Moore’s Law, Kryder’s Law and Kurzweil quoted in various articles, plus sources: [1] & [2]
[5] Conservative assumptions based upon Moore’s Law, Kryder’s Law and Kurzweil quoted in various articles, plus sources: [1] & [2]Tags: Assumption, Australia, Commissioner, Copenhagen, deep web, Denmark, Faculty of Advocates, France, free web, freely-available online publications, GBP, Information Commissioner's Office, Internet Preservation Consortium, Internet Watch Foundation, LDAP, New Zealand, Online Publications, online publishers, partial freely-available online publications, Permissions, printing, public relations, public relations overheads, Royal Library of Copenhagen, timely solution, United Kingdom, Web Archiving Consortium, web space