Web scraping

suggest change

Found a mistake? Have a question or improvement idea? Let me know.

Web scraping:

*Web scraping with Python

*Scraping using the Scrapy framework

*Basic example of using requests and lxml to scrape some data

*Maintaining web-scraping session with requests

*Scraping using Selenium WebDriver

*Scraping using BeautifulSoup4

*Modify Scrapy user agent

*Simple web content download with urllib.request

*Scraping with curl

Table Of Contents

0Getting Started

1List comprehension

2Filter

3List

4Functions

5Decorators

6Math module

7Loops

8Random module

9Comparisons

10Importing modules

11Sorting Minimum and Maximium

12Operator module

13Variable Scope and Binding

14Basic Input and Output

15Files, Folders, I/O

16JSON Module

17String Methods

18Metaclasses

19Indexing and Slicing

20Generators

21Simple Mathematical Operators

22Reduce

23Map Function

24Exponentation

25Searching

26Dictionary

27Classes

28Counting

29Manipulating XML

30Date and Time

31Set

32Collections module

33Parallel computation

34Multithreading

35Writing C extensions

36Unit Testing

37Regular Expressions

38Bitwise Operators

39Incompatibilities moving from Python 2 to Python 3

40Virtual environments

41Copying data

42Tuple

43Context Managers with Statement

44Hidden Features

45Enum

46String Formatting

47Conditionals

48Complex math

49Unicode and bytes

50The __name__ special variable

51Check if path exists

52Networking

53Asyncio Module

54Print Function

55os.path module

56Creating Python packages

57Parsing Command Line Arguments

58HTML Parsing

59Subprocess Library

60setup.py

61List slicing

62Sockets

63Itertools Module

64Recursion

65Boolean Operators

66dis module

67Type Hints

68pip PyPI Package Manager

69locale module

70Exceptions

71Web scraping

72deque module

73Distributing self-contained applications

74Property Objects

75Overloading

76Debugging

77Reading and Writing CSV

78Dynamic code execution with exec and eval

79PyInstaller - Distributing Python Code

80Iterables and Iterators

81Data Visualization

82The interpreter command line console

83args and kwargs

84functools module

85Garbage Collection

86Indentation

87Security and Cryptograhy

88Pickle data serialization

89urllib

90Binary Data

91Python and Excel

92Idioms

93Method Overriding

94Difference between a module and a package

95Data Serialization

96Python Concurrency

97RabbitMQ using AMQPStorm

98PostgreSQL

99Descriptor

100Common Pitfalls

101Multiprocessing

102Creating temporary files with tempfile

103Working with ZIP files

104Stack

105Profiling

106User-Defined Methods

107Working around Global Interpreter Lock

108Deployment using conda

109Logging

110Processes and Threads

111os module

112Comments and documentation

113Database Access

114Python HTTP Server

115Alternatives to switch statement from other languages

116List destructuring

117Accessing Python source code and bytecode

118Mixins

119Attribute Access

120ArcPy

121Python Anti-Patterns

122Plugin and Extension Classes

123Websockets

124Immutable data types

125String representations of class

126Arrays

127Operator Precedence

128Polymorphism

129Alternative Python implementations

130List Comprehensions

131Web Server Gateway Intrerface WSGI

1322to3 tool

133Abstract Syntax Tree

134Abstract Base Classes

135Unicode

136ssh in Python

137Serial Communication with pyserial

138Neo4j

139Performance optimization

140Curses

141Templates

142pass statement

143Testing with py.test

144Date Formatting

145heapq

146tkinter

147CLI subcommands

148Defining functions with list arguments

149SQLite3 module

150Persistence with pickle

151Connecting to SQL Server

152Design Patterns

153Multidimensional arrays

154Audio

155pyglet

156queue module

157ijson

158webbrowser module

159base64 module

160Flask

161Groupby

162Sockets and Message Encryption / Decryption

163pygame

164Input Subset and Output External Data Files using Pandas

165hashlib

166Gzip

167ctypes

168Creating a Windows Service

169Mutable vs. Immutable

170configparser

171Common Exceptions

172Optical Character Recognition OCR

173Python Data Types

174Partial functions

175Generating graphs

176Unzipping Files

177Functional Programming

178Python Virtual Environment - virtualenv

179sys module

180virtual environment with virtualenvwrapper

181virtualenvwrapper on Windows

182Python Requests Post

183Plotting with Matplotlib

184Python Lex-Yacc

185pyaudio

186shelve

187pip and PyPI Package Manager

188Writing to CSV from String or List

189Raise Custom Errors Exceptions

190Using loops within functions

191Contributors