Large File Upload: Implementing HTTP Resumable Uploads, Instant Uploads, and Chunked Uploads with Source Code!

Every evening at 18:00, let’s grow together!

Recently, due to the needs of the product in the R&D group I belong to, we required support for high-performance HTTP uploads of large files, with the capability for HTTP resumable uploads. Here’s a brief summary for easier memorization:

  1. The server side is implemented in C language, rather than using interpreted languages like Java or PHP;
  2. The server writes directly to the disk, thus avoiding the need to call move_uploaded_file or InputStreamReader, which require caching techniques to prevent server memory usage and browser request timeouts;
  3. Support for HTML5 and IFRAME (for older browsers), and the ability to retrieve file upload progress.

To better adapt to the current mobile internet, the upload service must support resumable uploads and reconnections, as mobile internet is not very stable; furthermore, the likelihood of an exception occurring during the upload of a large file is very high. To avoid re-uploading, supporting resumable uploads becomes very necessary.

The idea behind supporting resumable uploads is:

The client (usually the browser) uploads a file to the server and continuously records the upload progress. If a disconnection or other exception occurs, the client can query the server for the upload status of the file and continue uploading from the last uploaded position.

There are also experts online who use chunked file uploads to implement large file uploads, which involves splitting the file into smaller pieces, for example, 4MB per chunk. The server receives each small piece of the file and saves it as a temporary file, waiting for all chunks to be transmitted before executing a merge. I believe that if the original file is small enough, this method is feasible, but once the file is several hundred megabytes or several gigabytes or tens of gigabytes, the time to merge the files will be very long, often leading to browser response timeouts or server blocking.

If you implement an independent client (or a browser ActiveX plugin) to upload files, supporting resumable uploads will be very simple, as you only need to record the file upload status on the client side. However, supporting browser resumable uploads (without installing third-party plugins) is generally more challenging than creating an independent client upload, but it is not difficult. My implementation idea is as follows:

1. When the browser uploads a file, it first generates a HASH value for this file, which must be generated on the browser side.

It is not sufficient to rely solely on the file name to query the upload record, as file names can be very repetitive. The combination of file name + file size reduces the likelihood of repetition, and if you add the file modification time, the likelihood is further reduced. Adding a browser ID can further minimize conflicts. The best method for calculating the HASH value is to use the file’s content for MD5 calculation, but this is computationally intensive (and not really necessary), and excessive time consumption will affect the upload experience.

Based on the above reasons, my HASH value calculation idea is as follows:

  1. First, assign an ID to the browser, which is stored in a cookie;
  2. The result of the browser ID + file modification time + file name + file size is used to calculate a file HASH value using MD5;
  3. The browser ID is automatically granted to the browser when it accesses the file upload site.
// Simple cookie helper function
function setCookie(cname,cvalue,exdays)
{
var d = new Date();
  d.setTime(d.getTime()+(exdays*24*60*60*1000));
var expires = "expires="+d.toGMTString();
  document.cookie = cname + "=" + cvalue + "; " + expires;
}


function getCookie(cname)
{
var name = cname + "=";
var ca = document.cookie.split(';');
for(var i=0; i<ca.length; i++)
  {
    var c = ca[i].trim();
    if (c.indexOf(name)==0) return c.substring(name.length,c.length);
  }
return"";
}
//
// Simple file HASH value calculation, if you are not very particular, it should be usable for products.
// Since calculating the file HASH value uses various data, the possibility of HASH conflicts within the HYFileUploader system should be very small, and it should be safe to use.
// The ID of the file can be obtained using any algorithm, as long as it ensures that the ID for the same file is the same, and the length of the obtained ID should not exceed 32 bytes
//
function getFileId (file)
{
    // Grant a unique ID to the browser to distinguish different browser instances (different machines or the same machine with different browsers)
    var clientid = getCookie("HUAYIUPLOAD");
    if (clientid == "") {
        // Use a random value as the browser ID, which will be part of the file HASH value
        var rand = parseInt(Math.random() * 1000);
        var t = (new Date()).getTime();
        clientid =rand+'T'+t;

        setCookie("HUAYIUPLOAD",clientid,365);
    }

    var info = clientid;
    if (file.lastModified)
        info += file.lastModified;
    if (file.name)
        info += file.name;
    if (file.size)
        info += file.size;
    // https://cdn.bootcss.com/blueimp-md5/2.10.0/js/md5.min.js
    var fileid = md5(info);
    return fileid;
}

I believe: there is no need to read the content of the file to calculate the HASH value, as this would be very slow. If HTTP instant upload is indeed required, it may need to be done this way, so that if different people upload files with the same content, duplicate uploads can be avoided, and results can be returned directly.

The reason for assigning an ID to the browser is to further avoid HASH value conflicts for files with the same name and size from different computers.

2. Querying the file’s HASH value

In file upload support, first query the upload progress information from the upload server using the file’s HASH value, and then start uploading from the upload progress position. The code is as follows:

var fileObj = currentfile;
var fileid = getFileId(fileObj);
var t = (new Date()).getTime();
// Use the following URL to get the resumable upload information, the required parameter is fileid, appending the t parameter is to avoid browser caching
var url = resume_info_url + '?fileid='+fileid + '&t='+t;

var ajax = new XMLHttpRequest();

ajax.onreadystatechange = function () {
    if(this.readyState == 4){
        if (this.status == 200){
            var response = this.responseText;

            var result = JSON.parse(response);
            if (!result) {
                alert('The data returned by the server is incorrect, it may be an incompatible server');
                return;
            }
            // The file object returned by the resumable upload information contains the size already uploaded
            var uploadedBytes = result.file && result.file.size;
            if (!result.file.finished && uploadedBytes &lt; fileObj.size) {
                upload_file(fileObj,uploadedBytes,fileid);
            }
            else {
                // The file has been uploaded completely, no need to upload again, just return the result
                showUploadedFile(result.file);
                // Simulate progress completion
                //var progressBar = document.getElementById('progressbar');
                //progressBar.value = 100;
            }

        }else {
            alert('Failed to get file resumable upload information, please ask www.yoodb.com');
        }
    }
}

ajax.open('get',url,true);
ajax.send(null);

The above is the implementation using the jQuery-file-upload component. For the implementation code using raw JavaScript, please refer to the sample code in the demos directory h4resume.html.

3. Executing the upload

After querying the file’s resumable upload information, if the file has indeed been uploaded before, the server will return the size of the file that has been uploaded, and we can continue uploading data from the position of the already uploaded file size.

The HTML5 File object’s slice can be used to extract segments from the file for upload.

Definition and Usage

The slice() method can extract a portion of a file and return the extracted part as a new string.

Syntax

File.slice(start,end)

Parameters Description

start The starting index of the fragment to extract. If it is negative, this parameter specifies the position from the end of the string. That is, -1 refers to the last character of the string, -2 refers to the second to last character, and so on.

end The ending index of the fragment to extract. If this parameter is not specified, the substring to be extracted includes the string from start to the end of the original string.

If this parameter is negative, it specifies the position from the end of the string.

The code for implementing chunked file uploads is as follows:

/*
File upload processing code, www.yoodb.com
fileObj : html5 File object
start_offset: The starting position of the uploaded data relative to the file head
fileid: The ID of the file, which is obtained from the above getFileId function,
*/
function upload_file(fileObj,start_offset,fileid)
{
var xhr = new XMLHttpRequest();
var formData = new FormData();

var blobfile;

if(start_offset &gt;= fileObj.size){
return false;
 }

var bitrateDiv = document.getElementById("bitrate");
var finishDiv = document.getElementById("finish");
var progressBar = document.getElementById('progressbar');
var progressDiv = document.getElementById('percent-label');

var oldTimestamp = 0;
var oldLoadsize = 0;
var totalFilesize = fileObj.size;
if (totalFilesize == 0) return;

var uploadProgress = function (evt) {
if (evt.lengthComputable) {
   var uploadedSize = evt.loaded + start_offset;
   var percentComplete = Math.round(uploadedSize * 100 / totalFilesize);

   var timestamp = (new Date()).valueOf();
   var isFinish = evt.loaded == evt.total;

   if (timestamp &gt; oldTimestamp || isFinish) {
    var duration = timestamp - oldTimestamp;
    if (duration &gt; 500 || isFinish) {
     var size = evt.loaded - oldLoadsize;

     var bitrate = (size * 8 / duration /1024) * 1000; //kbps
     if (bitrate &gt; 1000)
      bitrate = Math.round(bitrate / 1000) + 'Mbps';
     else
      bitrate = Math.round(bitrate) + 'Kbps';

     var finish = evt.loaded + start_offset;

     if (finish &gt; 1048576)
      finish = (Math.round(finish / (1048576/100)) / 100).toString() + 'MB';
     else
      finish = (Math.round(finish / (1024/100) ) / 100).toString() + 'KB';

     progressBar.value = percentComplete;
     progressDiv.innerHTML = percentComplete.toString() + '%';
     bitrateDiv.innerHTML = bitrate;
     finishDiv.innerHTML = finish;

     oldTimestamp = timestamp;
     oldLoadsize = evt.loaded;
    }
   }
  }
else {
   progressDiv.innerHTML = 'N/A';
  }
 }

xhr.onreadystatechange = function(){
    if ( xhr.readyState == 4 && xhr.status == 200 ) {
      console.log( xhr.responseText );

    }
else if (xhr.status == 400) {

  }
  };

var uploadComplete = function (evt) {
  progressDiv.innerHTML = '100%';

var result = JSON.parse(evt.target.responseText);
if (result.result == 'success') {
   showUploadedFile(result.files[0]);
  }
else {
   alert(result.msg);
  }
 }

var uploadFailed = function (evt) {
  alert("File upload failed! For issues, please contact the public account: Java精选, ask questions");
 }

var uploadCanceled = function (evt) {
  alert("Upload was canceled or the browser disconnected!");
 }

// Set timeout, since it is a large file upload, do not set a timeout
//xhr.timeout = 20000;
//xhr.ontimeout = function(event){
//  alert('File upload time is too long, the server did not respond within the specified time!');
//}

 xhr.overrideMimeType("application/octet-stream");

var filesize = fileObj.size;
var blob = fileObj.slice(start_offset,filesize);
var fileOfBlob = new File([blob], fileObj.name);
// Additional file data should be placed at the front of the request
 formData.append('filename', fileObj.name);
// The fileid information must be sent to the server, the server will only perform resumable upload processing after obtaining the fileid information
 formData.append('fileid', fileid);
// The file data should be placed in the last field
//formData.append("file",blob, fileObj.name);
 formData.append('file', fileOfBlob);

 xhr.upload.addEventListener("progress", uploadProgress, false);

 xhr.addEventListener("load", uploadComplete, false);
 xhr.addEventListener("error", uploadFailed, false);
 xhr.addEventListener("abort", uploadCanceled, false);
 xhr.open('POST', upload_file_url);
//
 xhr.send(formData);
}

To verify file resumable uploads, I created a simple interface to display the status information during the file upload process, as shown below:

Large File Upload: Implementing HTTP Resumable Uploads, Instant Uploads, and Chunked Uploads with Source Code!

Stop using OFFSET and LIMIT pagination in MySQL, try this method!

Through HTML, you can calculate the upload progress of the file, the size of the file that has already been uploaded, the upload bitrate, and other information. If any exceptions occur during the upload process, you can re-upload, and the already uploaded parts will not need to be re-uploaded.

To verify HTML5 resumable uploads, you can download this file upload server from GitHub for testing.

Source code address: https://github.com/wenshui2008/UploadServer

Author:Da Fei Fei Yu

blog.csdn.net/ababab12345/article/details/80490621

The content published by the public account "Java精选" is noted as sourced, and the copyright belongs to the original source (those whose copyright cannot be verified or are not noted as sourced are from the internet, and are reprints, the purpose of reprinting is to convey more information, and the copyright belongs to the original author. If there is any infringement, please contact, and the author will delete and handle it immediately!)

Leave a Comment