![]() ![]() ![]() However, for files over 100M, almost all the files stored locally are inconsistent with the content-length in the obtained response header. ![]() Total_size = int(response.headers)įor chunk in er_content(chunk_size=102400):Īfter downloading, I will use the following method to compare with the above total_size with open('xxx', 'r') as f: Response = requests.get(url, stream=True) Relevant codes are summarized as follows: _RETRIES = 5 The probability is very high and should not be caused by network reasons. This generates unnecessary traffic and load on that server which is a pretty crappy thing to do The nicer and Pythonic thing to do is to have a separate script that does the request once and saves the required data to a local file. At present, I have a script to obtain files from the CDN chain as backup.Īt present, there is a situation that when using requests to download large files, the downloaded files will be incomplete. Every time you run that script with your requests.get code in place, you’re making a call to the target web server. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |