使用 R 语言进行网页抓取

数据科学领域最重要的事情之一是为您想要解决的问题获取正确数据的技能。数据科学家并不总是有一个准备好的数据库来处理，而是必须从正确的来源提取数据。为此，使用了 API 和 Web Scraping。

API（应用程序接口） ：API 是一组允许人们动态查询和检索数据的方法和工具。 Reddit、Spotify、Twitter、Facebook 和许多其他公司提供免费的 API，使开发人员能够访问他们存储在服务器上的信息；其他人则收取访问其 API 的费用。
网页抓取：很多数据无法通过数据集或 API 访问，而是作为网页存在于互联网上。因此，通过网络抓取，人们无需等待提供者创建 API 即可访问数据。

什么是网页抓取？

网页抓取是一种从网站获取数据的技术。在网上冲浪时，许多网站不允许用户保存数据以供私人使用。一种方法是手动复制粘贴数据，这既繁琐又耗时。 Web Scraping 是从网站中自动提取数据的过程。此过程是在称为网络抓取工具的网络抓取软件的帮助下完成的。他们根据用户要求自动从网站加载和提取数据。这些可以定制为适用于一个网站，也可以配置为适用于任何网站。

使用 R 实现 Web Scraping

有几种网络抓取工具可以执行任务，也有各种语言，有支持网络抓取的库。在所有这些语言中， R被认为是 Web Scraping 的编程语言之一，因为它具有丰富的库、易于使用、动态类型等特性。R 常用的 Web Scraping 工具是rvest 。

使用以下代码在您的 R Studio 中安装包rvest 。

install.packages('rvest')

拥有 HTML 和 CSS 知识将是一个额外的优势。据观察，大多数数据科学家对 HTML 和 CSS 的技术知识并不十分熟悉。因此，让我们使用一个名为Selector Gadget的开源软件，它对于任何人来说都足以执行 Web 抓取。可以访问和下载 Selector Gadget 扩展程序 (https://selectorgadget.com/)。考虑到按照网站上的说明安装了此扩展程序。另外，考虑一个使用谷歌浏览器的人，他/她可以访问右上角扩展栏中的扩展。

使用 rvest 在 R 中进行 Web 抓取

rvest 由传奇的 Hadley Wickham 维护。我们可以从这个库中轻松地从网页中抓取数据。

导入rvest库

在开始之前，我们会将 rvest 库导入到您的代码中。

R

library(rvest)

R

webpage = read_html("https://www.geeksforgeeks.org /\
data-structures-in-r-programming")

R

# Using CSS selectors to scrape the heading section
heading = html_node(webpage, '.entry-title')
 
# Converting the heading data to text
text = html_text(heading)
print(text)

R

# Using CSS selectors to scrape
# all the paragraph section
# Note that we use html_nodes() here
paragraph = html_nodes(webpage, 'p')
 
# Converting the heading data to text
pText = html_text(paragraph)
 
# Print the top 6 data
print(head(pText))

R

# R program to illustrate
# Web Scraping
 
# Import rvest library
library(rvest)
 
# Reading the HTML code from the website
webpage = read_html("https://www.geeksforgeeks.org /
data-structures-in-r-programming")
 
# Using CSS selectors to scrape the heading section
heading = html_node(webpage, '.entry-title')
 
# Converting the heading data to text
text = html_text(heading)
print(text)
 
# Using CSS selectors to scrape
# all the paragraph section
# Note that we use html_nodes() here
paragraph = html_nodes(webpage, 'p')
 
# Converting the heading data to text
pText = html_text(paragraph)
 
# Print the top 6 data
print(head(pText))

阅读 HTML 代码

使用read_html()从网页中读取 HTML 代码。考虑这个网页。

R

webpage = read_html("https://www.geeksforgeeks.org /\
data-structures-in-r-programming")

从 HTML 代码中抓取数据

现在，让我们从抓取标题字段开始。为此，请使用选择器小工具获取包含标题的特定 CSS 选择器。可以在他/她的浏览器中单击扩展名，然后用光标选择标题字段。

一旦知道包含标题的 CSS 选择器，他/她就可以使用这个简单的 R 代码来获取标题。

R

# Using CSS selectors to scrape the heading section
heading = html_node(webpage, '.entry-title')
 
# Converting the heading data to text
text = html_text(heading)
print(text)

输出：

[1] "Data Structures in R Programming"

现在，让我们抓取所有段落字段。为此，我们执行了与之前相同的程序。

一旦知道包含段落的 CSS 选择器，他/她就可以使用这个简单的 R 代码来获取所有段落。

R

# Using CSS selectors to scrape
# all the paragraph section
# Note that we use html_nodes() here
paragraph = html_nodes(webpage, 'p')
 
# Converting the heading data to text
pText = html_text(paragraph)
 
# Print the top 6 data
print(head(pText))

输出：

[1] “A data structure is a particular way of organizing data in a computer so that it can be used effectively. The idea is to reduce the space and time complexities of different tasks. Data structures in R programming are tools for holding multiple values. ”
[2] “R’s base data structures are often organized by their dimensionality (1D, 2D, or nD) and whether they’re homogeneous (all elements must be of the identical type) or heterogeneous (the elements are often of various types). This gives rise to the five data types which are most frequently utilized in data analysis. the subsequent table shows a transparent cut view of those data structures.”
[3] “The most essential data structures used in R include:”
[4] “”
[5] “A vector is an ordered collection of basic data types of a given length. The only key thing here is all the elements of a vector must be of the identical data type e.g homogeneous data structures. Vectors are one-dimensional data structures.”
[6] “Example:”

编程需要懂一点英语

使用 R 语言进行网页抓取的完整代码

R

# R program to illustrate
# Web Scraping
 
# Import rvest library
library(rvest)
 
# Reading the HTML code from the website
webpage = read_html("https://www.geeksforgeeks.org /
data-structures-in-r-programming")
 
# Using CSS selectors to scrape the heading section
heading = html_node(webpage, '.entry-title')
 
# Converting the heading data to text
text = html_text(heading)
print(text)
 
# Using CSS selectors to scrape
# all the paragraph section
# Note that we use html_nodes() here
paragraph = html_nodes(webpage, 'p')
 
# Converting the heading data to text
pText = html_text(paragraph)
 
# Print the top 6 data
print(head(pText))

输出：

[1] “Data Structures in R Programming”

[1] “A data structure is a particular way of organizing data in a computer so that it can be used effectively. The idea is to reduce the space and time complexities of different tasks. Data structures in R programming are tools for holding multiple values. ” [2] “R’s base data structures are often organized by their dimensionality (1D, 2D, or nD) and whether they’re homogeneous (all elements must be of the identical type) or heterogeneous (the elements are often of various types). This gives rise to the five data types which are most frequently utilized in data analysis. the subsequent table shows a transparent cut view of those data structures.” [3] “The most essential data structures used in R include:” [4] “” [5] “A vector is an ordered collection of basic data types of a given length. The only key thing here is all the elements of a vector must be of the identical data type e.g homogeneous data structures. Vectors are one-dimensional data structures.” [6] “Example:”

编程需要懂一点英语