Node.JS Streams — File
In this tutorial I’ll show different ways to response files to your client from your back-end, and explain the benefits of use Node.JS steams.
First, we’ll see the response using only “send" method. Then we can compare it with the stream method.
You could check all used code here.
📢 Important Informations:
System Info: *this information is about my system, where I run the tests
File Info:
This is the information about the file that will be send to the client.
File Size: 147MB
📏 Monitoring memory usage
To see the behavior of our application, well’ll monitor the usage of memory. To do that, every 5 seconds the total of memory usage will be logged at our console.
We could do that using this code:
const processUsage = process.memoryUsage();
const totalUsedMemoryInMb = Math.floor(processUsage.rss / (1024 * 1024));
console.log(`Used Memory: ${totalUsedMemoryInMb} MB`);This is the result of the code above:
As you can see, 40MB is the base consumption of application when startup. By the way, note that we are using typescript and development mode, when we build the application this consumption will decrease. But for our test purpose, it’s ok, we just need to guarantee that all test will be run at same environment.
📤 Making the requests
Send method
First let’s analyse using the res.send method to response our file. You can see the implementation:
app.get('/send-sync', (req, res) => {
const filePath = resolve(__dirname, '..', 'upload', 'ex.mp4');
fs.readFile(filePath, (err, data) => {
res.send(data);
});
}This is an inefficient way to send files, because this loads the entire file into application memory before response to the client. If we make 10 requests to this route, the file will be loaded 10 times in memory, even thought it’s the same file.
We can see this happening at the screenshot below. I opened 10 tabs at my browser wheres load a html that renders the file using the route above.
As you can see, the consumption of memory increased strongly, using a total of 880 MB. Remember that the video has 140MB! So, if you want scalability, this certainly not the best implementation to use.
Streams method
We know that Node make things in small chunks. To response information to the client, it do that sending little pieces of data without storage in memory. We could use this approach with our file too, sending the file directly of our file system in small pieces. This process is called Streaming.
You can see the implementation of this approach:
app.get('/stream', (req, res) => {
const { range } = req.headers;
const filePath = resolve(__dirname, '..', 'upload', 'ex.mp4');
const { size: fileSize } = fs.statSync(filePath); const start = Number(range?.replace(/\D/g, '')) || 0;
const CHUNK_SIZE = 1000000;
const end = Math.min(start + CHUNK_SIZE, fileSize - 1); const headers = {
'Content-Range': `bytes ${start}-${end}/${fileSize}`,
'Accept-Ranges': 'bytes',
'Content-Type': 'video/mp4',
'Content-Length': CHUNK_SIZE,
}; res.writeHead(206, headers); const fileStreams = fs.createReadStream(filePath, { start, end });
fileStreams.pipe(res);
}
Briefly, the code get information about the request to know witch part of the file should response. Then start to read this part of the file and simultaneously sending to client.
To test it, opened 10 tabs again:
You can see the big difference of memory consumption. With the same number of requests, now it’s almost imperceptible the usage of memory. This happens because the file system doesn’t use the memory to read the file.
Streams using sendFile method
Alternatively we can use sendFile method to response the file using stream approach. It works similar at the method explained above, but has slight difference in usability, mainly due customization.
You could check the code below:
app.get('/send-file', (req, res) => {
const filePath = resolve(__dirname, '..', 'upload', 'ex.mp4');
return res.sendFile(filePath);
}The implementation is simpler and perform similar as “stream hard implementation”. You could see at test:
It performs similar as “stream hard method”, the memory usage is almost the same. And to scale your application, you can use this simple implementation without loose performance.
As I said, this two implementations have slight differences. If want an article about that, comment below 😁
You can run those tests too! Check the repository below and run! At README you have the full instructions to make it!
Connect with me:
