Home » Python » Downloading Files Using Python (Simple Examples)

About Mokhtar Ebrahim

Mokhtar Ebrahim
LikeGeeks: Learn more about Linux, Server Administration, Web Development, Python, iOS, and Tech Tips

Downloading Files Using Python (Simple Examples)

In this tutorial, you will learn how to download files from the web using different Python modules. You will download regular files, web pages, YouTube videos, Google drive files, Amazon S3, and other sources.

Also, you will learn how to overcome many challenges that you may counter such as downloading files that redirects,  downloading large files, multithreaded download, and other tactics.

Using requests

You can download files from a URL using the requests module.

Consider the code below:

1
2
3
4
5
6
7
import requests
 
 
myfile = requests.get(url)
 
open('c:/users/LikeGeeks/downloads/PythonImage.png', 'wb').write(myfile.content)

Simply, get the URL using the get method of requests module and store the result into a variable named “myfile”. Then you write the contents of the variable into a file.

Using wget

You can also download a file from a URL by using the wget module of Python. The wget module can be installed using pip as follows:

pip install wget

Consider the following code in which we will download the logo image of Python:

1
2
3
4
5
import wget
 
 
wget.download(url, 'c:/users/LikeGeeks/downloads/pythonLogo.png')

In this code, the URL along with the path (where the image will be stored) is passed to the download method of the wget module.

Download file that redirects

In this section, you will learn to download from a URL which redirects to another URL with a .pdf file using requests. The URL is like the following:

https://readthedocs.org/projects/python-guide/downloads/pdf/latest/

To download this pdf file, use the following code:

1
2
3
4
5
6
7
import requests
 
 
myfile = requests.get(url, allow_redirects=True)
 
open('c:/users/LikeGeeks/documents/hello.pdf', 'wb').write(myfile.content)

In this code, the first step we specify the URL. Then we use the get method of the requests module to fetch the URL. In the get method, we set the allow_redirects to True which will allow redirection in the URL and the content after redirection will be assigned to the variable myfile.

Finally, we open a file to write the fetched content.

Download large file in chunks

Consider the code blew:

01
02
03
04
05
06
07
08
09
10
11
12
13
import requests
 
url = 'https://www.cs.uky.edu/~keen/115/Haltermanpythonbook.pdf'
 
r = requests.get(url, stream = True)
 
with open("PythonBook.pdf", "wb") as Pypdf:
 
    for chunk in r.iter_content(chunk_size = 1024):
 
        if chunk:
 
            Pypdf.write(chunk)

First, we use the get method of the requests module as we did before but this time we will set the stream attribute to True.

Then we create a file named PythonBook.pdf in the current working directory and open it for writing.

Then we specify the chunk size that we want to download at a time. We have set to 1024 bytes. Iterate through each chunk and write the chunks in the file until the chunks finished.

The Python shell will look like the following when the chunks are downloading:

Python modules

Not pretty? Don’t worry, we will show a progress bar for the downloading process later.

Download multiple files (Parallel/bulk download)

To download multiple files at a time, import the following modules:

1
2
3
4
5
6
7
import os
 
import requests
 
from time import time
 
from multiprocessing.pool import ThreadPool

We imported the os and time modules to check how much time it takes to download files. The module ThreadPool lets you run multiple threads or processes using the pool.

Let’s create a simple function which sends the response to a file in chunks:

01
02
03
04
05
06
07
08
09
10
11
def url_response(url):
 
    path, url = url
 
    r = requests.get(url, stream = True)
 
    with open(path, 'wb') as f:
 
        for ch in r:
 
            f.write(ch)

The URLs is a two-dimensional array which specifies the path and the URL of a page you want to download.

Pass the URL to requests.get as we did in the previous section. Finally, open the file (path specified in the URL) and write the content of the page.

Now we can call this function for each URL separately and we can also call this function for all the URLs at the same time. Let’s do it for each URL separately in for loop and notice the timer:

1
2
3
4
5
6
7
start = time()
 
for x in urls:
 
    url_response (x)
 
print(f"Time to download: {time() - start}")

The result will be like this:

Python modules

Now replace the for loop with the following line of code:

ThreadPool(9).imap_unordered(url_response, urls)

Run the script:

Python modules

Download with a progress bar

The Progress bar is a UI widget of the clint module. To install the clint module, type the following command:

pip install clint

Python modules

Consider the following code:

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
import requests
 
from clint.textui import progress
 
 
r = requests.get(url, stream=True)
 
with open("LearnPython.pdf", "wb") as Pypdf:
 
    total_length = int(r.headers.get('content-length'))
 
    for ch in progress.bar(r.iter_content(chunk_size = 2391975), expected_size=(total_length/1024) + 1):
 
        if ch:
 
            Pypdf.write(ch)

In this code, we imported the requests module and then from clint.textui we imported the progress widget. The only difference is in the for loop. We used the bar method of progress module while writing the content into the file. The output will be like the following:

Python modules

Download a webpage using urllib

In this section, we will be downloading a webpage using the urllib.

The urllib library is a standard library of Python so you do not need to install it.

The following line of code can easily download a webpage:

urllib.request.urlretrieve(‘url’, ‘path’)

Specify the URL here that you want to save as and where you want to store it:

urllib.request.urlretrieve(‘https://www.python.org/’, ‘c:/users/LikeGeeks/documents/PythonOrganization.html’)

In this code, we used the urlretrieve method and passed the URL of a file along with the path where we will save the file. The file extension will be .html.

Download via proxy

If you need to use a proxy to download your files, you can use the ProxyHandler of the urllib module. Check the following code:

1
2
3
4
5
6
7
import urllib.request
 
>>> myProxy = urllib.request.ProxyHandler({'http': '127.0.0.2'})
 
>>> openProxy = urllib.request.build_opener(myProxy)
 
>>> urllib.request.urlretrieve('https://www.python.org/')

In this code, we created the proxy object and opened the proxy by invoking the build_opener method of urllib and passed the proxy object. Then we made the request to retrieve the page.

Python modules

Also, you can use the requests module as documented in the official documentation:

1
2
3
4
5
import requests
 
myProxy = { 'http': 'http://127.0.0.2:3001' }
 
requests.get("https://www.python.org/", proxies=myProxy)

Simply import the requests module and create your proxy object. Then you can retrieve the file.

Python modules

Using urllib3

The urllib3 is an improved version of the urllib module. You can download and install it using pip:

pip install urllib3

We will fetch a web page and store it in a text file by using urllib3.

Import the following modules:

import urllib3, shutil

The shutil module is used when working with files.

Now initialize the URL string variable like this:

url = ‘https://www.python.org/’

Then we use the PoolManager of urllib3 that keeps track of necessary connection pools.

c = urllib3.PoolManager()

Create a file:

filename = “test.txt”

Finally, we send a GET request to fetch the URL and open a file and write the response into that file:

1
2
3
with c.request('GET', url, preload_content=False) as res, open(filename, 'wb') as out_file:
 
    shutil.copyfileobj(res, out_file)

Download from Google drive

You can use googledrivedownloader to download any file from google drive. To install it, use the following command:

pip install googledrivedownloader

Check out the code below:

1
2
3
4
5
from google_drive_downloader import GoogleDriveDownloader as gd
 
gd.download_file_from_google_drive(file_id='0B7XV2PwnZyfNalJ6cFd6dXBrckE',
 
dest_path='./data/2ndHalfJava.zip', unzip=True)

Here we passed the file id of the Google drive file along with the destination path where we will save the file. Then we have the unzip parameter. If it is True, the downloaded file will be unzipped in the same destination folder.

Python modules

In this example, we download the zip folder then the folder is unzipped.

Download file from S3 using boto3

To download files from Amazon S3, you can use the Python boto3 module.

Before getting started, you need to install the awscli module using pip:

pip install awscli

Python modules

For AWS configuration, run the following command:

aws configure

Now enter your details as:

1
2
3
4
5
6
7
AWS Access Key ID [None]: (The access key)
 
AWS Secret Access Key [None]: (Secret access key)
 
Default region name [None]: (Region)
 
Default output format [None]: (Json)
Python modules

To download a file from Amazon S3, import boto3 and botocore. Boto3 is an Amazon SDK for Python to access Amazon web services such as S3. Botocore provides the command line services to interact with Amazon web services.

Botocore comes with awscli. To install boto3 run the following:

pip install boto3

Python modules

Now import these two modules:

import boto3, botocore

When downloading files from Amazon, we need three parameters:

  1. The name of Bucket
  2. The name of the file you need to download
  3. The name of the file after it has downloaded

Initialize the variables:

1
2
3
4
5
bucket = "bucketName"
 
file_name = "filename"
 
downloaded_file = "downloadedfilename"

Now initialize a variable to use the resource of a session. For this, we will call the resource() method of boto3 and pass the service which is s3:

service = boto3.resource(‘s3’)

Finally, download the file by using the download_file method and pass in the variables:

service.Bucket(bucket).download_file(file_name, downloaded_file)

Download videos from Youtube

To download videos from YouTube, you can use the pytube module.

First, install the pytube module using pip as:

pip install pytube

Python modules

Now, import YouTube from pytube as:

from pytube import YouTube

Then pass the video URL you need to download:

YouTube(‘https://www.youtube.com/watch?v=himEMfYQJ1w’).streams.first().download()

In this line of code, we passed the URL. Then there are streams (list of formats) that the video has. For example:

1
2
3
4
5
streams = yt.streams.all()
 
>>> for x in streams:
 
    print(x)

This will print all formats along with the video resolution as follows:

01
02
03
04
05
06
07
08
09
10
11
<Stream: itag="22" mime_type="video/mp4" res="720p" fps="30fps" vcodec="avc1.64001F" acodec="mp4a.40.2">
<Stream: itag="43" mime_type="video/webm" res="360p" fps="30fps" vcodec="vp8.0" acodec="vorbis">
<Stream: itag="18" mime_type="video/mp4" res="360p" fps="30fps" vcodec="avc1.42001E" acodec="mp4a.40.2">
<Stream: itag="36" mime_type="video/3gpp" res="240p" fps="30fps" vcodec="mp4v.20.3" acodec="mp4a.40.2">
<Stream: itag="17" mime_type="video/3gpp" res="144p" fps="30fps" vcodec="mp4v.20.3" acodec="mp4a.40.2">
<Stream: itag="136" mime_type="video/mp4" res="720p" fps="30fps" vcodec="avc1.4d401f">
<Stream: itag="135" mime_type="video/mp4" res="480p" fps="30fps" vcodec="avc1.4d4014">
<Stream: itag="134" mime_type="video/mp4" res="360p" fps="30fps" vcodec="avc1.4d401e">
<Stream: itag="133" mime_type="video/mp4" res="240p" fps="30fps" vcodec="avc1.4d400c">
<Stream: itag="160" mime_type="video/mp4" res="144p" fps="30fps" vcodec="avc1.4d400b">
<Stream: itag="140" mime_type="audio/mp4" abr="128kbps" acodec="mp4a.40.2">

The first() method grabs the first format and the download() method downloads the video.

If you want to fetch information about a video, for example, the title, then use .title as follows:

1
2
3
4
5
 
>>> video.title
 
'Linux Environment Variables'

Using asyncio

The asyncio module is focused on handling system events. It works around an event loop that waits for an event to occur and then reacts to that event. The reaction can be calling another function. This process is called even handling. The asyncio module uses coroutines for event handling.

To use the asyncio event handling and coroutine functionality, we will import the asyncio module:

import asyncio

Now define the asyncio coroutine method like this:

1
2
3
async def coroutine():
 
    await my_func()

The keyword async tells that this is a native asyncio coroutine. Inside the body of the coroutine, we have the await keyword which returns a certain value. The return keyword can also be used.

Now let’s create a code using a coroutine to download a file from the web:

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
>>> import os
 
>>> import urllib.request
 
>>> async def coroutine(url):
 
    r = urllib.request.urlopen(url)
 
    filename = "couroutine_downloads.txt"
 
    with open(filename, 'wb') as f:
 
        for ch in r:
 
            f.write(ch)
 
    print_msg = 'Successfully Downloaded'
 
    return print_msg
 
>>> async def main_func(urls_to_download):
 
    co = [coroutine(url) for url in urls_to_download]
 
    downloaded, downloading = await asyncio.wait(co)
 
    for i in downloaded:
 
        print(i.result())
 
 
 
 
 
>>> eventLoop = asyncio.get_event_loop()
 
>>> eventLoop.run_until_complete(main_func(urls_to_download))

In this code, we created an async coroutine function that downloads our files and returns a message.

Then we have another async coroutine calls the main_func that waits for the URLs and make a queue of all URLs. The wait function of asyncio waits for the coroutines to complete.

Now to start the coroutine, we have to put the coroutine inside the event loop by using the get_event_loop() method of asyncio and finally, the event loop is executed using the run_until_complete() method of asyncio.

Downloading files using Python is fun. I hope you find the tutorial useful.

Thank you.

Published on Web Code Geeks with permission by Mokhtar Ebrahim, partner at our WCG program. See the original article here: Downloading Files Using Python (Simple Examples)

Opinions expressed by Web Code Geeks contributors are their own.

(0 rating, 0 votes)
You need to be a registered member to rate this.
Start the discussion Views Tweet it!
Do you want to know how to develop your skillset to become a Web Rockstar?
Subscribe to our newsletter to start Rocking right now!
To get you started we give you our best selling eBooks for FREE!
1. Building web apps with Node.js
2. HTML5 Programming Cookbook
3. CSS Programming Cookbook
4. AngularJS Programming Cookbook
5. jQuery Programming Cookbook
6. Bootstrap Programming Cookbook
and many more ....
I agree to the Terms and Privacy Policy

Leave a Reply

avatar

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  Subscribe  
Notify of