网页抓取

面向垂直搜索引擎的网页抓取器的设计和实现

Vertical Search Engine for Crawling the Web Page Design and Implementation

介绍了网页抓取、正文提取和词语切分的预处理过程。

Pre-procedure was introduced , including page crawler , content extraction and word segmentation .

并行网页抓取系统设计

Design of a Parallel Web Crawling System

该系统采用模块化设计，实现了从网页抓取到聚类的整个网页聚类过程。

The system is modular in design , implementation of the cluster from a web page to process the entire web page clustering .

页逐页检索可以通过使用网页抓取内容源，但这只能是与公共门户网站的页面上使用。

Page-by-page crawling can be achieved by using the web crawler content source but this can only be used with public Portal pages .

并通过实验证明，该方法具有网页抓取的高效性以及页面分类的准确性。

The experiments verified efficiency of web crawling and accuracy of pages classification . Additionally , we describe an incremental update crawler system in Deep Web .

第二章阐述如何利用目前流行的网页抓取、分析及数据库构建技术构建检索的后台数据源。

The second chapter illustrates how to use currently popular technologies including web page grabbing , web page analysis and database construction to build retrieval data source .

通过网页抓取、网页清洗和数据存储构建分类语料库，并在此基础之上利用不同特征选择算法和分类算法实现了自动归类。

After the construction of a self-build classification corpus , four features selection algorithms have been used with the classification algorithm Simple Vector Distance to finish automatic classification .

在网页的抓取过程中，利用MD5摘要算法实现了对重复的URL和内容相同的Web页面的排除，并提出了摘要算法的替代方案。

In process of fetching web page , the system used MD5 algorithm to remove the repeated URLs and the web page with same content , and also put forward a substitution solution for digest algorithm .

在此基础上，结合《中国农业网站名录》中收录的6000余个网址，开发了网页自动抓取工具，将抓回的网页利用SDD算法建立起语义索引，成功构建了一个中文农业搜索引擎。

Using more than 6000 URLs , the tool of automatically web grabber has been developed then the web pages was indexed with the specific agricultural dictionary , finally a specific search engine of agriculture was made .

论文完成的主要工作如下：（1）实现了对旅游突发事件网页的抓取和分析。

The main job and innovation are as follows : ( 1 ) Crawl and analyze the tourism emergency pages .

在此基础上，使之能够进行主题网页的抓取和判断，实现真正的面向主题的搜索。

On this basis , it can grab and judge the subject pages , and then realizes the true Subject-oriented Search Engine .

主题爬虫是主题搜索引擎的信息采集部分，负责对用户感兴趣的某一主题的网页进行抓取。

Focused crawler is information collection part of focused search engine and it fetches some topic web pages in which users are interest .

结合程序代码说明，一步一步地完成对指定网页的抓取、产品参数信息的抽取、生成词库、建立索引和将信息保存到数据库。

Combining with the explanation of program code , step by step , finish crawling pages , extracting information of product parameters , generating product word Library , constructing product indexer and saving the information into database .

对于海洋科学数据文件的下载，本文设计了一个专用的网页文件抓取器并提出了海洋数据元数据文件的提取算法，从而能够有效的抓取到海洋数据文件并进行正确的解析。

For getting the marine scientific data , we design a special web crawler and give an algorithm to extract metadata files . In this way , we could not only download these ocean data files on Web , but also understand their meaning .

本文主要研究网页数据的抓取和解析，网页数据的抓取是由网络蜘蛛Spider完成的，而网页数据的解析是指从抓取到的网页中提取出结构化的信息。

Crawling the web data is done by spider . Web data analysis refers to extracting structured information from the crawled web pages .

网页超链抓取及自动分类技术实现

A Scheme of Extraction Hyperlink from Web Page and Automatic Classification

首先对基本的网页分析算法进行分析综述：如基于广度优先策略和最佳优先策略的网页抓取方法。

Firstly , the basic algorithm to analyze web analytics Summary : If breadth-first strategy based on priority strategies and best way to crawl web pages .