最近在研究如何爬取数据,所以使用pyspider 来完成,不过比较坑的是mac从python2.7 到python 3.0 需要重新安装很多的库。

尝试使用如下命令:

pip3 install pyspider

安装后直接报错:

Curl is configured to use SSL, but we have not been able to determine which SSL backend it is using. Please see PycURL documentation for how to specify the SSL backend manually

英文的意思是:Curl依赖的SSL没有配置正确,当然网上有很多解决方案,所以通过互联网发现有很多。这里博主给总结下解决办法:

第一步:升级下pip:pip3 install --upgrade pip
第二步:

export LDFLAGS=-L/usr/local/opt/openssl/lib
export CPPFLAGS=-I/usr/local/opt/openssl/include
export PYCURL_SSL_LIBRARY=openssl

第三步:pip3 install pycurl

Collecting pycurl
  Using cached https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz
Building wheels for collected packages: pycurl
  Building wheel for pycurl (setup.py) ... done
  Stored in directory: /Users/landsnail/Library/Caches/pip/wheels/d2/85/ae/ebf5ff0f1368869d082b4863df492bf54c661bf6306a2bdfde
Successfully built pycurl
Installing collected packages: pycurl
Successfully installed pycurl-7.43.0.2

安装成功!!!!

最后安装一下pyspider,pip3 install pyspider

Successfully installed Flask-1.0.3 Flask-Login-0.4.1 Jinja2-2.10.1 MarkupSafe-1.1.1 PyYAML-5.1 Werkzeug-0.15.4 click-7.0 defusedxml-0.6.0 itsdangerous-1.1.0 jsmin-2.2.2 pyspider-0.3.10 tblib-1.4.0 tornado-4.5.3 u-msgpack-python-2.5.1 wsgidav-3.0.0

至此完美解决!