News - Portal spiders hard at work
Portal spiders hard at work
September 2002
As reported in our last newsletter, agencies have described over 1000 services and twice as many associated resources like brochures and forms with high-quality metadata. So far they have only described services they think people are most likely to want to find easily through the portal. And rather than try to describe every single page on their websites, agencies have described collections of pages. This gets you quickly to the place on the Education Review Office site where school education evaluation reports can be found, for example.
The portal will help you find a great deal more. Like the current NZGO site, the portal is already ‘spidering’ sites to build a huge index of the material deep within government websites. This includes
- All .govt.nz sites (central and local government) and .mil.nz (defence forces)
- Those .co.nz and .org.nz sites which we know to be part of government, such as acc.co.nz
- All .cri.nz (Crown Research Institutes)
- The home pages only of universities, polytechs and wananga.
Website spidering will be continuous, catching updates to sites efficiently and regularly. In addition, the portal operators will be able to run special purpose, targeted spiders to ensure that significant information is updated in a timely way, as is done with events such as the annual Budget release.
Unlike real spiders, the portal’s virtual spiders don’t work frenetically. They request pages at the leisurely rate of two pages a minute, so have little or no impact on agency web servers.
