see, scrapy

common

• fetch(url)
• shelp()
• help(<response>): help about response

[s] Available Scrapy objects:
[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s]   Out        {2: <200 https://docs.scrapy.org/en/latest/topics/shell.html>, 3: Type help() for interactive help, or help(object) for help about object., 5: True, 7: [<Selector query='//h2' data='<h2>Configuring the shell<a class="he...'>, <Selector query='//h2' data='<h2>Launch the shell<a class="headerl...'>, <Selector query='//h2' data='<h2>Using the shell<a class="headerli...'>, <Selector query='//h2' data='<h2>Example of shell session<a class=...'>, <Selector query='//h2' data='<h2>Invoking the shell from spiders t...'>], 8: [<Selector query='//h2/text()' data='Configuring the shell'>, <Selector query='//h2/text()' data='Launch the shell'>, <Selector query='//h2/text()' data='Using the shell'>, <Selector query='//h2/text()' data='Example of shell session'>, <Selector query='//h2/text()' data='Invoking the shell from spiders to in...'>]}
[s]   _2         <200 https://docs.scrapy.org/en/latest/topics/shell.html>
[s]   _oh        {2: <200 https://docs.scrapy.org/en/latest/topics/shell.html>, 3: Type help() for interactive help, or help(object) for help about object., 5: True, 7: [<Selector query='//h2' data='<h2>Configuring the shell<a class="he...'>, <Selector query='//h2' data='<h2>Launch the shell<a class="headerl...'>, <Selector query='//h2' data='<h2>Using the shell<a class="headerli...'>, <Selector query='//h2' data='<h2>Example of shell session<a class=...'>, <Selector query='//h2' data='<h2>Invoking the shell from spiders t...'>], 8: [<Selector query='//h2/text()' data='Configuring the shell'>, <Selector query='//h2/text()' data='Launch the shell'>, <Selector query='//h2/text()' data='Using the shell'>, <Selector query='//h2/text()' data='Example of shell session'>, <Selector query='//h2/text()' data='Invoking the shell from spiders to in...'>]}
[s]   crawler    <scrapy.crawler.Crawler object at 0x10eaf6eb0>
[s]   item       {}
[s]   request    <GET https://docs.scrapy.org/en/latest/topics/shell.html#topics-shell>
[s]   response   <200 https://docs.scrapy.org/en/latest/topics/shell.html>
[s]   settings   <scrapy.settings.Settings object at 0x10eb2f160>
[s]   spider     <DefaultSpider 'default' at 0x11092a3d0>
[s] Useful shortcuts:
[s]   fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s]   fetch(req)                  Fetch a scrapy.Request and update local objects
[s]   shelp()           Shell help (print this help)
[s]   view(response)    View response in a browser

configuration

Scrapy will look for configuration parameters in ini-style scrapy.cfg files in standard locations:

• /etc/scrapy.cfg or c:\scrapy\scrapy.cfg (system-wide),
• ~/.config/scrapy.cfg ($XDG_CONFIG_HOME) and ~/.scrapy.cfg ($HOME) for global (user-wide) settings, and
• scrapy.cfg inside a Scrapy project’s root (see next section).

ipython 安装

• pip3 install ipython
• scrapy shell: launch

skill

• play with splash: jump

link

• topics-shell (opens new window)