The ‘Scrapy: Powerful Web Scraping & Crawling with Python’ course will teach you how to scrape websites and build a powerful web crawler using Scrapy, Splash and Python. The course also teaches how to Integrate Splash with Scrapy to scrape JavaScript rendered websites.
The course teaches how to create a web crawler in Scrapy and also you will learn how to deploy & schedule Spiders to ScrapingHub. The course is usually available for INR 2,599 on Udemy but you can click on the link and get the ‘Scrapy: Powerful Web Scraping & Crawling with Python’ for INR 499.
Who all can opt for this course?
- This Scrapy tutorial is designed for individuals who are already familiar with Python and want to learn how to build a web crawler and scraper that is effective for navigating websites and extracting content from pages that contain useful information
Course Highlights
| Key Highlights | Details |
|---|---|
| Registration Link | Apply Now! |
| Price | INR 499 ( |
| Duration | 10.5 Hours |
| Rating | 4.4/5 |
| Student Enrollment | 16,133 students |
| Instructor | GoTrained Academy https://www.linkedin.com/in/gotrainedacademy |
| Topics Covered | Web Scraping, Python, Xpath, API |
| Course Level | Advanced |
| Total Student Reviews | 2,726 |
Learning Outcomes
- Use Scrapy to build a web crawler
- Scraping data by crawling one or more pages
- Spider deployment and scheduling on ScrapingHub
- Scrapy’s use for website logins
- Scrapy being run as a standalone script
- Integrating Splash with Scrapy to scrape websites that use JavaScript
- Scraping JavaScript-driven web pages with Scrapy and Selenium in Particular Situations
- Constructing an advanced scrapy spider
- Further features that Scrapy provides when Spider has finished scraping
- Using Scrapy Parameters for Editing
- Creating files in CSV, Excel, XML, or JSON from data scraped by Scrapy
- Putting data that Scrapy has extracted into databases like MySQL and MongoDB
- Several actual web scraping projects, including those for Craigslist, LinkedIn, and numerous more websites
- You can download the Python source code for each exercise in this Scrapy tutorial
- Use the Q&A board to post your inquiries and immediately receive responses
Course Content
| S.No. | Module (Duration) | Topics |
|---|---|---|
| 1. | Scrapy vs. Other Python Web Scraping Frameworks (11 minutes) | Scrapy vs. Beautiful Soup vs. Selenium |
| Course Tips (Must Read) | ||
| 2. | Scrapy Installation (10 minutes) | Linux Scrapy Installation |
| Mac Scrapy Installation | ||
| Windows Scrapy Installation | ||
| Scrapy Installation Instructions | ||
| Python Editor: Sublime Text | ||
| 3. | Building Basic Spider with Scrapy (34 minutes) | Scrapy Simple Spider – Part 1 |
| Scrapy Simple Spider – Part 2 | ||
| Scrapy Simple Spider – Part 3 | ||
| 4. | XPath Syntax (08 minutes) | Using XPath with Scrapy |
| Tools to Easily Get XPath | ||
| 5. | Q&A (28 seconds) | Scrapy Basics |
| Do you have questions so far? | ||
| XPath Syntax | ||
| 6. | Building More Advanced Spider with Scrapy (38 minutes) | Scrapy Advanced Spider – Part 1 |
| Scrapy Advanced Spider – Part 2 | ||
| Scrapy Advanced Spider – Part 3 | ||
| Scrapy Advanced Spider – Part 4 | ||
| Scrapy Architecture | ||
| 7. | Web Scraping Best Practices (01 minutes) | Avoid Getting Banned! |
| 8. | Deploying & Scheduling Scrapy Spider on ScrapingHub (10 minutes) | ScrapingHub: Deploying & Scheduling Scrapy Spiders (UPDATED) |
| 9. | Logging into Websites Using Scrapy (17 minutes) | Logging into Websites Using Scrapy |
| 10. | Scrapy as a Standalone Script (UPDATED) (02 minutes) | Scrapy as a Standalone Script (UPDATED) |
| 11. | Building Web Crawler with Scrapy (10 minutes) | Building Web Crawler with Scrapy |
| 12. | Scrapy with Selenium (23 minutes) | Why/When We Should Use Selenium |
| Selenium WebDriver + Scrapy Selector to Extract URLs | ||
| Selenium Loading Next for Data Extraction (usable even with JavaScript pages) | ||
| Getting Data | ||
| 13. | Scrapy with Splash – JavaScript Websites (01 hour 03 minutes) | Splash Prerequisite: Install Docker (NEW) |
| Splash Installation (NEW) | ||
| How to use Splash with Scrapy (NEW) | ||
| Splash Advanced Project: Scraping Baierl.com p.1 (NEW) | ||
| Splash Advanced Project: Scraping Baierl.com p.2 (NEW) | ||
| Splash Advanced Project: Scraping Baierl.com p.3 (NEW) | ||
| 14. | Scrapy Spider – Bookstore (26 minutes) | Grabbing URLs |
| Data Extraction | ||
| 15. | More about Scrapy (15 minutes) | Scrapy Arguments |
| Scrapy Close Function | ||
| Scrapy Items | ||
| 16. | Export Output to Files (29 minutes) | Scrapy Feed Exports to CSV, JSON, or XML |
| Export Output to Excel | ||
| Downloading Images with Scrapy Pipelines | ||
| Renaming Images with Scrapy Pipelines | ||
| 17. | Scrapy Project #1: Scraping Craigslist Eng Jobs in NY (51 minutes) | Craigslist Scraper – Overview |
| Creating Scrapy Craigslist Spider | ||
| Craigslist Scrapy Spider #1 – Titles | ||
| Craigslist Scrapy Spider #2 – One Page | ||
| Craigslist Scrapy Spider #3 – Multiple Pages | ||
| Craigslist Scrapy Spider #4 – Job Descriptions | ||
| Editing Scrapy settings.py (e.g. throttling, user agent, etc.) | ||
| Final Scrapy Tutorial, Craigslist Spider Code | ||
| 18. | Extracting Data to Databases – MySQL & MongoDB (39 minutes) | Installing MySQL |
| MySQL Installation and Usage | ||
| Writing Data to MySQL | ||
| Installing MongoDB | ||
| MongoDB Installation and Usage | ||
| Writing Data to MongoDB | ||
| 19. | Scrapy Project #2: Web Scraping Class-Central.com (32 minutes) | Scraping Class-Central – Part 1: Subjects (UPDATED) |
| Scraping Class-Central – Part 2: Courses (UPDATED) | ||
| 20. | Scrapy Advanced Topics (59 minutes) | Scrapy User Agent |
| Scraping Tables (UPDATED) | ||
| Scraping JSON Pages | ||
| Scrapy FormRequest (UPDATED) | ||
| Using Multiple Proxies with Crawlera (Optional) | ||
| 21. | Scrapy Project #3: Web Scraping Dynamic Website eplanning.ie (01 hour 01 minutes) | ePlanning Scraping Project Overview |
| ePlanning: Extracting Initial URLs | ||
| ePlanning: Crawling Internal Pages | ||
| ePlanning: Scrapy Form Requests | ||
| ePlanning: Scraping Data | ||
| ePlanning: Checking Data Existence | ||
| ePlanning: Scraping Data from Table | ||
| 22. | Project #4: Scraping Shoes’ Prices from API Request (36 minutes) | Scraping Product Prices from API Request p.1 (NEW) |
| Scraping Product Prices from API Request p.2 (NEW) | ||
| Scraping Product Prices from API Request p.3 (NEW) | ||
| 23. | Project #5: Web Scraping LinkedIn.com (UPDATED) (01 hour 00 minutes) | LinkedIn Scraping Project: Overview & Requirements (UPDATED) |
| LinkedIn Logging in (UPDATED) | ||
| Finding LinkedIn Profiles: Part 1 (UPDATED) | ||
| Finding LinkedIn Profiles: Part 2 (UPDATED) | ||
| Scraping Data Points from LinkedIn Profiles: Part 1 (UPDATED) | ||
| Scraping Data Points from LinkedIn Profiles: Part 2 (UPDATED) | ||
| Connecting to LinkedIn Profiles (UPDATED) | ||
| 24. | Solved Web Scraping Exercises (02 minutes) | Yield Data Items from 2 Functions |
| How to Order Exported Data | ||
| Xpath contains() and starts-with() functions | ||
| 25. | Bonus: Data Extraction with APIs (17 seconds) | Data Extraction with APIs (Free Tutorial) |
| 26. | Bonus: Web Scraping with Beautiful Soup, Requests & Selenium Course (01 minutes) | Coupon for Web Scraping with Beautiful Soup, Requests & Selenium & Other Courses |
Resources Required
- This Scrapy course presupposes that you are already familiar with the fundamentals of developing straightforward Python programmes and with the language’s essential capabilities (data structures, file handling, functions, classes, modules, common libraries, etc)
- Python 2.7 or Python 3.3
- The best operating system is any one (Linux, Mac, Windows)
- Positive attitude, openness to learning new things, and willingness to ask questions (if any) during the course’s Q&A session
- Please read the course description and watch the preview lectures Before enrolling in the course if you have any questions about what Scrapy is or why you should use it.
Featured Review
Umar Ibrahim (4/5) : the course is excellent for me and it has provided me the best of what I need to be an expert in data scraping with python
Pros
- Andre Da Rocha (5/5) : This course is by bar the best on teaching how to use scrapy and the basics of web scraping.
- Qingling Ni (4/5) : This course is probably the best to learn how to use Scrapy.
- ATM Sayfuddin (5/5) : The instructor responds promptly and the best thing about this course is that I can progress through my project as I learn during the course.
- Anjaneya Prasad Nidubrolu (5/5) : I felt the course is awesome and tutor is at his best.
Cons
- Yang Chenjie (1/5) : If I have no prior knowledge, I can hardly understand what the instructor is talking about.
- Anonymized User (1/5) : None of the commands work in windows, was not aware that a prerequisite for this course was to have a Ubuntu Linux computer.
- Abhishek Tripathi (1/5) : While the course has alot of content, it doesn’t explain what certain things mean and how they work. Q&A is Slow aswell, its been 2 days since i asked a question and i did not get a answer yet, had to buy another course from someone else just to understand how to solve my problem
About the Author
The instructor of this course is GoTrained Academy who is a eLearning Professionals. With 3.7 Instructor Rating and 6,365 Reviews on Udemy, GoTrained Academy offers 9 Courses and has taught 74,624 Students so far.
- GoTrained is an e-learning academy that focuses on technology and management while producing helpful information in several languages
- GoTrained Academy use a unique method for choosing the content we offer, they primarily concentrate on abilities that are regularly required by clients and jobs even though there aren’t many videos that teach them
- GoTrained Academy also make an effort to create video series that cover both the fundamentals and more advanced topics
Comparison Table
| Parameters | Scrapy: Powerful Web Scraping & Crawling with Python | Advanced Web Scraping with Python using Scrapy & Splash | Modern Web Scraping with Python using Scrapy Splash Selenium |
|---|---|---|---|
| Offers | INR 499 ( | INR 455 ( | INR 455 ( |
| Duration | 10.5 hours | 5.5 hours | 9 hours |
| Rating | 4.4/5 | 4.8/5 | 4.4/5 |
| Student Enrollments | 16,133 | 6,160 | 22,790 |
| Instructors | GoTrained Academy | Ahmed Rafik | Ahmed Rafik |
| Register Here | Apply Now! | Apply Now! | Apply Now! |
Leave feedback about this