Scrapy is a very helpful and powerful open-source web crawling framework. It is written in Python and designed with efficiency and scalable web scraping in mind. With Scrapy, you can extract data from websites, process the data if required, and store it in different formats and databases. The tech community uses MongoDB with Scrapy. MongoDB is a NoSQL database, familiar for its flexibility and scalability. When you combine these two tools together, you have a great framework, capable of handling large-scale web scraping projects. Let's get started with our Scrapy tutorial right away!
How to Set Up Scrapy
Before we get to the part where we set up Scrapy, let us see the prerequisite first. Scrapy needs Python. As a precaution, ensure you have Python installed in your system. Scrapy is the preferred tool among all Python web scrapers. Then, you can set up Scrapy by executing this pip command:
pip install scrapy
Now that we have installed Scrapy, let us create our new Scrapy project. To create a Scrapy project, execute this command:
scrapy startproject myproject
Executing this command creates a project directory with all the required files and folders. In this directory, you can create spiders (more on spiders in the next section).
Create a Scrapy Spider
Spiders are classes that define how to crawl a website and extract data. In other words, it is how you instruct Scrapy how to scrape data from a website. Here is a simple spider example that extracts quotes from a website:
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'http://quotes.pythoncentral.io/',
]
def parse(self, response):
for quote in response.css('div.quote'):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('small.author::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
}
next_page = response.css('li.next a::attr(href)').get()
if next_page is not None:
yield response.follow(next_page, self.parse)
When this script is run, our spider starts its job at the specified URL, extracts the required data using CSS classes, and follows pagination links to continue scraping until there are no more pages to scrape.
How to Integrate Scrapy with MongoDB
Now that we have scraped data with Python i.e., with the help of spider, we have to store the data. To store the data, let us use MongoDB database and then configure Scrapy to connect to it. For doing this, let us install the "pymongo" library. Execute this command to install the pymongo library:
pip install pymongo
Our next step is to create an item pipeline that processes and stores the scraped data. Here is a sample for you to get started:
import pymongo
class MongoPipeline:
def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db
@classmethod
def from_crawler(cls, crawler):
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DATABASE', 'items')
)
def open_spider(self, spider):
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]
def close_spider(self, spider):
self.client.close()
def process_item(self, item, spider):
self.db[spider.name].insert_one(dict(item))
return item
Now, navigate to your Scrapy project's settings and add these configurations:
ITEM_PIPELINES = {
'myproject.pipelines.MongoPipeline': 300,
}
MONGO_URI = 'mongodb://localhost:27017'
MONGO_DATABASE = 'scrapy_data'
With this setup, you can ensure that each item extracted by our spider is now stored in the specified MongoDB database.
Common Challanges in Python Web Scraping
Web scraping becomes difficult when we have to do scraping dynamic websites. When it comes to managing request rates and avoiding IP bans, there are some precautions we can take to overcome these challenges. Let us see some tricks to overcome the challenges in handling dynamic content with Scrapy:
- User-agent rotation: By randomizing the user-agent header, you can prevent servers from identifying and blocking your spiders from scraping their websites.
- Request throttling: By adjusting the download delay between consecutive requests you can reduce the load on target servers. This method reduces the risk of getting blocked.
- JavaScript handling: When you are working with sites that rely heavily on JS, integrate Scrapy with automation tools like Selenium or Splash.
By using these configurations, you can amplify the efficiency and reliability of your Python web scraping projects.
Key Takeaways
Combining Scrapy with MongoDB provides a robust solution for web scraping and data storage. Scrapy's powerful crawling capabilities, paired with MongoDB's flexible schema design, allow for efficient extraction and storage of large datasets. By understanding how to set up and configure these tools, you can streamline your data collection processes and focus on analysing the information that matters most.
By mastering Scrapy and MongoDB, you can develop efficient and scalable web scraping solutions tailored to your data collection needs.