标签搜索
侧边栏壁纸
博主昵称
梦之中小俊

以前高中时，羡慕大学考试只要及格就行；现在大学了，怀念高中考试及不及格都行??

python

py爬虫-已解决requests.exceptions.ConnectionError: (‘Connection aborted .’, ConnectionResetError(10054，“远程主机强迫关

梦之中小俊
梦之中小俊
2024-09-25 / 0 评论 / 594 阅读 / 推送成功！
报错原因


报错原因：

1.、http的连接数超过最大限制，默认的情况下连接是Keep-alive的，所以这就导致了服务器保持了太多连接而不能再新建连接。

2、ip被封

3、程序请求速度过快。

粉丝爬虫爬了太多页了，没有挂代码被别人网站识别出爬虫了小伙伴按下面的方法设置代码IP和请求头即可解决！！！

def get_url(url):
    try:
        response = requests.get(url, headers=headers, timeout=10)  # 超时设置为10秒
    except:
        for i in range(4):  # 循环去请求网站
            response = requests.get(url, headers=headers, timeout=20)
            if response.status_code == 200:
                break
    html_str = response.text
    return html_str
​

第二种解决方法：构建headers列表每次随机选一个：

headers_list = [
    {
        'user-agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 10; SM-G981B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (iPad; CPU OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/87.0.4280.77 Mobile/15E148 Safari/604.1'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.109 Safari/537.36 CrKey/1.54.248666'
    }, {
        'user-agent': 'Mozilla/5.0 (X11; Linux aarch64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.188 Safari/537.36 CrKey/1.54.250320'
    }, {
        'user-agent': 'Mozilla/5.0 (BB10; Touch) AppleWebKit/537.10+ (KHTML, like Gecko) Version/10.0.9.2372 Mobile Safari/537.10+'
    }, {
        'user-agent': 'Mozilla/5.0 (PlayBook; U; RIM Tablet OS 2.1.0; en-US) AppleWebKit/536.2+ (KHTML like Gecko) Version/7.2.1.0 Safari/536.2+'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; U; Android 4.3; en-us; SM-N900T Build/JSS15J) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; U; Android 4.1; en-us; GT-N7100 Build/JRO03C) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; U; Android 4.0; en-us; GT-I9300 Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 7.0; SM-G950U Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 8.0.0; SM-G965U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 8.1.0; SM-T837A) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.80 Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; U; en-us; KFAPWI Build/JDQ39) AppleWebKit/535.19 (KHTML, like Gecko) Silk/3.13 Safari/535.19 Silk-Accelerated=true'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; U; Android 4.4.2; en-us; LGMS323 Build/KOT49I.MS32310c) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/102.0.0.0 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Windows Phone 10.0; Android 4.2.1; Microsoft; Lumia 550) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Mobile Safari/537.36 Edge/14.14263'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 6.0.1; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 6.0.1; Nexus 10 Build/MOB31T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 4.4.2; Nexus 4 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 8.0.0; Nexus 5X Build/OPR4.170623.006) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 7.1.1; Nexus 6 Build/N6F26U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 8.0.0; Nexus 6P Build/OPP3.170518.006) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 6.0.1; Nexus 7 Build/MOB30X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (compatible; MSIE 10.0; Windows Phone 8.0; Trident/6.0; IEMobile/10.0; ARM; Touch; NOKIA; Lumia 520)'
    }, {
        'user-agent': 'Mozilla/5.0 (MeeGo; NokiaN9) AppleWebKit/534.13 (KHTML, like Gecko) NokiaBrowser/8.5.0 Mobile Safari/534.13'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 9; Pixel 3 Build/PQ1A.181105.017.A1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 10; Pixel 4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 11; Pixel 3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.181 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (Linux; Android 8.0.0; Pixel 2 XL Build/OPD1.170816.004) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Mobile Safari/537.36'
    }, {
        'user-agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1'
    }, {
        'user-agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1'
    }, {
        'user-agent': 'Mozilla/5.0 (iPad; CPU OS 11_0 like Mac OS X) AppleWebKit/604.1.34 (KHTML, like Gecko) Version/11.0 Mobile/15A5341f Safari/604.1'
    }
]

headers = random.choice(headers_list)

第三种方法解决方法：放慢请求速度

time.sleep(10)
​

第四种方法解决方法：爬虫代码设置代理ip


proxies = {
        'http': '127.0.0.1:1212',
        'https': '127.0.0.1:1212'
    }
r = requests.get(url, headers=headers, proxies=proxies, timeout=20)
​

本文共 个字数，平均阅读时长 ≈ 分钟，您已阅读：0时0分0秒。
3

打赏

注意：本站资源多为网络收集，如涉及版权问题请及时与站长联系，我们会在第一时间内删除资源。
本站用户发帖仅代表本站用户个人观点，并不代表本站赞同其观点和对其真实性负责。
本站资源大多存储在云盘，如发现链接失效，请及时与站长联系，我们会第一时间更新。
转载本网站任何内容，请按则和转载方式正确书写本站原文地址
版权属于： 梦之中小俊
本文链接： https://api.layacms.com/wb/article/293.html
作品采用： 《 署名-非商业性使用-相同方式共享 4.0 国际 (CC BY-NC-SA 4.0) 》许可协议授权
相关推荐
PHP的异常处理
小黄鸟微信抓包工具Windows版本 HttpCanary 和Reqable
 PHP的异常处理
mysql优化分页排序
 PHP的异常处理
MySQL优化之 EXPLAIN 详解：全面解析执行计划
 PHP的异常处理
MySQL EXPLAIN 详解

评论 (0)

OωO
  • ::(呵呵)
  • ::(哈哈)
  • ::(吐舌)
  • ::(太开心)
  • ::(笑眼)
  • ::(花心)
  • ::(小乖)
  • ::(乖)
  • ::(捂嘴笑)
  • ::(滑稽)
  • ::(你懂的)
  • ::(不高兴)
  • ::(怒)
  • ::(汗)
  • ::(黑线)
  • ::(泪)
  • ::(真棒)
  • ::(喷)
  • ::(惊哭)
  • ::(阴险)
  • ::(鄙视)
  • ::(酷)
  • ::(啊)
  • ::(狂汗)
  • ::(what)
  • ::(疑问)
  • ::(酸爽)
  • ::(呀咩爹)
  • ::(委屈)
  • ::(惊讶)
  • ::(睡觉)
  • ::(笑尿)
  • ::(挖鼻)
  • ::(吐)
  • ::(犀利)
  • ::(小红脸)
  • ::(懒得理)
  • ::(勉强)
  • ::(爱心)
  • ::(心碎)
  • ::(玫瑰)
  • ::(礼物)
  • ::(彩虹)
  • ::(太阳)
  • ::(星星月亮)
  • ::(钱币)
  • ::(茶杯)
  • ::(蛋糕)
  • ::(大拇指)
  • ::(胜利)
  • ::(haha)
  • ::(OK)
  • ::(沙发)
  • ::(手纸)
  • ::(香蕉)
  • ::(便便)
  • ::(药丸)
  • ::(红领巾)
  • ::(蜡烛)
  • ::(音乐)
  • ::(灯泡)
  • ::(开心)
  • ::(钱)
  • ::(咦)
  • ::(呼)
  • ::(冷)
  • ::(生气)
  • ::(弱)
  • ::(狗头)
泡泡
阿鲁
颜文字
取消
  1. 头像
    6767 Lv.1
    iPhone · Safari
    沙发

    画图

    回复
  2. 头像
    6767 Lv.1
    Windows 10 · Google Chrome
    板凳

    表情

    回复
  3. 头像
    透露 Lv.2
    Android · QQ Browser
    第30楼

    649494848

    回复