Django uses csv, HttpResponse, Streaming HttpResponse to export

Posted by dennisflash on Sun, 01 Sep 2019 17:09:19 +0200

First: Preface

In django, how to download csv files of website data to users? We have learned httpResponse before and used it together.

Case 1 - Small File Export

1. Case description
We provide the information in the database directly to the client to download.

  1. Add database model and insert data;
  2. New view, using HttpResponse, csv module to return data, generate csv format to return;
  3. New url mapping;
    In this case, this method can only solve the problem of small amount of data, if the amount of data is very large, it may be unable to export due to server timeout.

2. Database Code

from django.db import models

# Create your models here.

class Article(models.Model):
    id = models.BigAutoField(primary_key=True)
    title = models.CharField(max_length=100)
    content = models.TextField()
    price = models.FloatField(default=0)
    create_time = models.DateTimeField(auto_now_add=True)

    class Meta:
        db_table = 'k1_method_article'

#Using makegrations to generate migration scripts
#python manage.py makemigrations
# Mapping newly generated script files to databases
# python manage.py migrate

3. View Code

def csv_downloads1(request):
    data = []  #List data to be returned eventually
    db_result = Article.objects.all()
    for row in db_result:
        source_data = []  #Generate a list of each row of information in the database
        source_data.append(row.title)
        source_data.append(row.content)
        source_data.append(row.price)
        data.append(source_data)  #Add each row of data to a large list
    response = HttpResponse(content_type='text/csv') #Define an HttpResponse of type CSV
    response['Content-Disposition'] = "attachment;filename=huangjaijin.csv" #Define the return information, attachment mode and file name;
    writer = csv.writer(response)  #Write response using csv module
    writer.writerow(['Title', 'content', 'Price']) #Heading line
    for row in data:
        writer.writerow(row)  #Loop write database information
    return response

3. URL code

urlpatterns = [
    path('csv_downloads1/', views.csv_downloads1, name='csv_downloads1'),
]

4. URL code
Database insertion data

5. Access and download
http://127.0.0.1:8000/k02_httpresponse/csv_downloads1/

Case 2 - Generating csv files

def csv_downloads2(request):
    data = [
        ['Name', 'Height'],
        ['Keys', '176cm'],
        ['HongPing', '160cm'],
        ['WenChao', '176cm']
    ]
    with open('templates/abc.csv', 'w', encoding='utf-8', newline='') as f:
        writer = csv.writer(f, dialect='excel')
        for row in data:
            writer.writerow(row)
    return HttpResponse("Success")

Case 3 - Export of Big Documents

Assuming that we need to export a 1000W row data csv file, the method of using Case 1 is to generate data, then csv writes to response (waiting to write), after writing, the web page returns to download; then in the writing process, it may be due to server timeout exception. Is there a way to download while generating? We can use the Streaming HttpResponse object for implementation. The code is as follows:
(1) Using Streaming HttpResponse

def large_csv_downloads(request):
    # Using Streaming HttpResponse
    response = StreamingHttpResponse(content_type='text/csv')
    response['Content-Disposition'] = "attachment;filename=huangjaijin.csv"
    result = ("Row {}, {} \n".format(row, row) for row in range(0, 10000000))   #generator
    '''
    print(type(result)): <class 'generator'>
    print(result):  <generator object large_csv_downloads.<locals>.<genexpr> at 0x0000000004512660>
    '''
    response.streaming_content = result
    return response

Note: Generator Case Explanation

In [38]: L = [x*2 for x in range(5)]

In [39]: L
Out[39]: [0, 2, 4, 6, 8]

In [40]: G = (x*2 for x in range(5))  #The difference is that [] becomes ()

In [41]: G
Out[41]: <generator object <genexpr> at 0x7fad5268d910>

In [42]: for num in G:
    ...:     print(num)
    ...:     
0
2
4
6
8

Download address: http://127.0.0.1:8000/k02_httpresponse/large_csv_downloads

We can see that when we open the web page, we start to download at the moment, because it is downloading while generating data.

(2) Common csv and HttpResponse
The case here is that we download the same typed files in a normal way and try.

def large_csv_downloads(request):
    #Using HttpResponse
    response = HttpResponse(content_type='text/csv')
    response['Content-Disposition'] = "attachment;filename=huangjaijin.csv"
    writer = csv.writer(response)
    for row in range(0, 10000000):
        writer.writerow(['Row {}'.format(row), ''.format(row)])
    return response

We can find that when we open a web page, it's just a link waiting.

Topics: Web Development Database Django Python encoding