Array Buffers in JavaScript

In this article, we look at Array Buffers and how they are used to handle raw binary data in JavaScript.

In JavaScript, an ArrayBuffer is defined as an object that represents a generic, fixed-length raw binary data buffer. It allows you to work with binary data directly without worrying about character encoding. This means the ArrayBuffer handles raw binary data, such as working with network protocol files.

To understand it better without using too many unfamiliar words, ArrayBuffer is used to keep and arrange binary data. It handles raw binary data, such as working with network protocol files and manipulating raw data.

Think of the ArrayBuffer as a box of chocolates. The box size is fixed and can only contain a certain number of chocolates; regardless of whether any chocolates are in it, the size is fixed. The box still takes up space and has a defined capacity; it cannot be compromised, shrunk or grown. The box is the ArrayBuffer, while the chocolates are the binary data.

Uses of ArrayBuffer

ArrayBuffer is used to handle raw binary data effectively. It is particularly useful when dealing with network protocols, file input and output (I/O), and other situations where you need to work with binary data.

The goal when using ArrayBuffer is to to seamlessly handle raw binary data during processes, for cases such as :

• Fast data processing
• Handling files and network data
• Web APIs interactions
• Games, graphics and other high-quality visual data

Creating an ArrayBuffer

To create an array buffer, you only need the ArrayBuffer constructor and the size (in bytes).

Here is how it works:

const buffer = new ArrayBuffer(8); //creates a buffer of 8 bytes
console.log(buffer.byteLength); //output: 8

In the example above, we created a fixed-size block of memory with 8 bytes. It is currently empty because it does not contain any usable data.

Working with Binary Data (Typed Arrays)

As explained in the introduction, ArrayBuffer doesn’t allow direct manipulation, but that doesn’t mean it can’t be manipulated. To interact with an ArrayBuffer, you need typed arrays like Uint8Array, Int16Array or Float32Array, or you can use DataView for more flexible manipulation.

Typed arrays are a special kind of array that allows you to store and manipulate raw binary data in a specific format like 8-bit integers, 16-bit floats and more. They only allow one type of data and have a fixed size.

Here are the list of types:

• Int8Array: 8-bit signed integers (1 byte, range: -128 to 127)
• Uint8Array: 8-bit unsigned integers (1 byte, range: 0 to 255)
• Uint8ClampedArray: 8-bit clamped unsigned integers (1 byte, range: 0 to 255, but values outside this range are “clamped” to the nearest valid value)
• Int16Array: 16-bit signed integers (2 bytes, range: -32,768 to 32,767)
• Uint16Array: 16-bit unsigned integers (2 bytes, range: 0 to 65,535)
• Int32Array: 32-bit signed integers (4 bytes, range: -2,147,483,648 to 2,147,483,647)
• Uint32Array: 32-bit unsigned integers (4 bytes, range: 0 to 4,294,967,295)
• Float32Array: 32-bit floating-point numbers (4 bytes, range: approximately -3.4 × 10^38 to 3.4 × 10^38)
• Float64Array: 64-bit floating-point numbers (8 bytes, range: approximately -1.8 × 10^308 to 1.8 × 10^308)

The choice of the type to be used depends on these factors:

1. The type of data you are storing: If you are working with whole numbers, use Int8Array, Uint8Array, Uint8ClamedArray, Int16Array, Uint16Array, Int32Array or Uint32Array. If you are working with decimal numbers, use Float32Array or Float64Array.
2. Range of values: If you’re working with negative values, use a signed type like Int8Array, Int16Array, etc. If you’re working with only positive values, use an unsigned type like Uint8Array, Uint16Array, etc.
3. Precision and memory efficiency: For your system memory’s sake, always consider the smallest type that can safely store your value. If the smallest type can’t work, you can now go for the next bigger one. For numbers ranging from 0 to 255, always consider Uint8Array before Uint32Array because it’s more memory efficient.
4. Special use cases: If you are working with color data like pixels, such as images, Uint8Array or Uint8ClampedArray is used. For audio and video data, Float32Array is commonly used.

Let’s manipulate the array buffer we created in the previous example (remember it is empty) using a typed array of Uint8Array, Int16Array and Float32Array.

const buffer = new ArrayBuffer(8); //creates a buffer of 8 bytes

Using Uint8Array

const uint8view = new Uint8Array(buffer); //creates a view as 8-bit unsigned integers

uint8view[0] = 65; //store value at index 0
uint8view[1] = 66; //store value at index 1

console.log(uint8view); //Output: Uint8Array(8) [65, 66, 0, 0, 0, 0, 0, 0]

Using Int16Array

const int16View = new Int16Array(buffer); //creates a view as 16-bit unsigned integers

int16View[0] = 300; // store 300 at byte 0,
int16View[1] = -500; // store -500 at byte 1

console.log(int16View); // Output: Int16Array(4) [300, -500, 0, 0]

Using Float32Array

const float32View = new Float32Array(buffer);

float32View[0] = 3.14; //store a floating-point number at byte 0,

console.log(float32View); //Output: Float32Array(2) [ 3.140000104904175, 0 ]

Easy Reading and Writing (DataView)

Unlike typed arrays, which only allow one type of data and a fixed size, DataView allows you to read and write different types of data sizes, making it much easier to work with ArrayBuffer.

Here is a list of types in DataView:

• setInt8(offset, value): 8-bit signed integers
• setUint8(offset, value): 8-bit unsigned integers
• setInt16(offset, value): 16-bit signed integers
• setUint16(offset, value): 16-bit unsigned integers
• setInt32(offset, value): 32-bit signed integers
• setUint32(offset, value): 32-bit unsigned integers
• setFloat32(offset, value): 32-bit floating point
• setFloat64(offset, value): 64-bit floating point

The offset is the index or position of a byte in the array buffer. See it as a slot to uniquely store a byte in a buffer while value is the byte to be stored.

Let’s see how to use DataView.

Creating an ArrayBuffer and DataView

const buffer = new ArrayBuffer(8); //Create an 8-byte memory block
const view = new DataView(buffer); //Create a DataView for easy reading/writing

In the code above, a new ArrayBuffer with 8 bytes of memory is created. It is initially empty and is stored in a variable name buffer. We then used DataView to read and write different data types for the ArrayBuffer.

Writing Data to the Buffer

view.setInt8(0, 100); // Store the value 100 at the first byte (position 0) as an 8-bit signed integer
view.setUint16(1, 500); // Store the value 500 at the next two bytes (positions 1 and 2) as a 16-bit unsigned integer
view.setFloat32(3, 3.14); // Store the value 3.14 at the next four bytes (positions 3 to 6) as a 32-bit floating-point number
console.log(view); // Output the DataView object showing the total byte length (8 bytes)

In the code above, we store three different data types in the buffer: setInt8, setUint16 and setFloat32 using DataView.

setInt8(0, 100) has an offset of 0 and a value of 100; it stores 100 at byte position 0. setInt8 can either be positive or negative, but 100 is positive.

setUint16(1, 500) has an offset of 1 and a value of 500. It stores 500 in byte positions 1 and 2 (2 bytes). This is because in binary, 500 is 00000001 11110100. The first byte takes position 1, while the second byte takes position 2.

setFloat32(3, 3.14) has an offset of 3 and a value of 3.14; it stores 3.14 in byte position 3 to 6 (4 bytes). This is because of how floating point numbers are represented in the computer. Float32 uses the IEEE 754 standard and takes 4 bytes according to the computer. In binary, 3.14 is 01000000 01001100 11110011 11010101. The first byte takes position 1, the second byte takes position 2, the third byte takes position 3, and the fourth byte takes position 4.

All these are different data types stored in a buffer, which can only be achieved using DataView.

Reading Data from the Buffer

console.log(view.getInt8(0)); // 100
console.log(view.getUint16(1)); // 500
console.log(view.getFloat32(3)); // 3.14

In the code above, we are reading the code of different data types stored in the buffer and getting them according to their offset.

Converting Strings to and from ArrayBuffer

JavaScript provides several methods to convert between strings and ArrayBuffers, which is essential for tasks such as encoding, decoding and managing data streams. We can’t talk about conversion between strings and ArrayBuffer without talking about TextEncoder and TextDecoder.

Convert a String to ArrayBuffer

To convert a string to an ArrayBuffer, you typically need to encode the string into a binary format. The most common encoding is UTF-8.

Here is how to achieve it:

function stringToArrayBuffer(str) {
  let encoder = new TextEncoder();
  return encoder.encode(str).buffer;
}

let myString = "Hello, World";
let buffer = stringToArrayBuffer(myString);
console.log(buffer); //ArrayBuffer containing UTF-8 encoded string.

The code above is a function for converting a string to an array buffer using TextEncoder.

Convert an ArrayBuffer to a String

To convert an ArrayBuffer to a string, you need to decode the binary data. The TextDecoder object is used to decode an ArrayBuffer back to a string.

function arrayBufferToString(buffer) {
  let decoder = new TextDecoder();
  return decoder.decode(new Uint8Array(buffer));
}

let decodedString = arrayBufferToString(buffer);
console.log(decodedString); //"Hello, World"

In the code above, we used the function to convert the arrayBuffer to string using TextDecoder.

Handling Different Encodings

While UTF-8 is the most common encoding, sometimes you might need to handle other encodings. The TextEncoder and TextDecoder can be used with different encodings by simply specifying the encoding type as shown below:

let encoder = new TextEncoder("utf-16");
let buffer = encoder.encode("Hello, World!");
let string = decoder.decode(buffer);
console.log(string); //"Hello, World"

Fetching Binary Data Using ArrayBuffer

Web APIs often require data to be in a binary format. For instance, when using the Fetch API to download binary files, the response is an ArrayBuffer. Converting this binary data to a string might be necessary for further processing or display.

fetch("example.com/data")
  .then((response) => response.arrayBuffer())
  .then((buffer) => {
    let text = arrayBufferToString(buffer);
    console.log(text);
  });

Application of ArrayBuffer

ArrayBuffer is important when dealing with performance-intensive tasks, especially when they involve binary data.

Here are some of its applications in modern software:

1. File-sharing apps or file handling (images, videos, audio, PDFs, etc.): Web applications often process large files, which can sometimes slow down the computer or cause system lag. With the help of ArrayBuffer, this process is handled efficiently without slowing down.

let fileInput = document.querySelector('input[type="file"]');
fileInput.addEventListener("change", function (event) {
  let file = event.target.files[0];
  let reader = new FileReader();
  reader.onload = function () {
    let arrayBuffer = reader.result;
    let text = arrayBufferToString(arrayBuffer);
    console.log(text);
  };
  reader.readAsArrayBuffer(file);
});

In the code above, input[type="file"] is an HTML tag for uploading a file. The file is read as an array buffer and later converted to a string to be displayed in the console.

2. Streaming applications like Netflix or YouTube that process large data: The best way to do this is to process it in chunks instead of loading it all at once. ArrayBuffer helps manage and handle this.
3. In Web APIs interactions: Fetch API, WebSocket, Socket.IO and more work with raw binary data. ArrayBuffer also helps provide smooth sending and receiving of this data.
4. Online gaming: Gaming platforms are performance-driven applications. They use libraries like game engines and 3D rendering, and these libraries use ArrayBuffer to handle textures, models and animations efficiently.
5. Biometric authentication: Array buffers are heavily used to represent passkeys, which allows users to authenticate on applications using their biometrics rather than regular passwords. The structure of the payload received when passkeys are created or used to authenticate is in the form of array buffers.

Best Practices and Performance Considerations

It is clear that ArrayBuffer is important for managing high-performance applications. However, these best practices should be considered for optimal use:

1. Don’t abuse DataView: When working with only one data type, use TypedArray, which is faster than DataView. Avoid DataView unless you need to work with mixed data types.
2. Avoid unnecessary duplication of ArrayBuffer: ArrayBuffer is stored in memory and, when created multiple times, can waste memory. Try to avoid array buffer copies as much as possible. When working with parts of a buffer, use subarray() instead of slice() when possible.
3. Always clear the memory when the ArrayBuffer is no longer needed. ArrayBuffer can’t be directly tracked in memory, especially if it’s large; you can manually clear or nullify its references to free up memory.

Here is how to do that:

let buffer = new ArrayBuffer(8);
buffer = null; // Memory Cleared!

4. Use streams to fetch large binary files: Loading a large chunk of binary data at once consumes memory and can cause the main thread to freeze. Use streams to process data in chunks. An example of this stream is response.body.getReader().

Here is the code explaining this:

async function fetchLargeFile(url) {
  const response = await fetch(url);
  const reader = response.body.getReader();

  const chunks = [];
  let receivedLength = 0;

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    chunks.push(value);
    receivedLength += value.length;
    console.log(`Received ${receivedLength} bytes so far...`);
  }
  // Combine all chunks into a single Uint8Array
  const fullData = new Uint8Array(receivedLength);
  let position = 0;
  for (const chunk of chunks) {
    fullData.set(chunk, position);
    position += chunk.length;
  }
  console.log("Download complete!");
  return fullData;
}
// Example usage
fetchLargeFile("https://example.com/large-file.bin").then((data) => {
  console.log("Final file size:", data.length);
});

We created an async function called fetchLargeFile which accepts a URL as an argument. We use getReader to read the files in small chunks instead of downloading them at once. We create an array called chunks to collect and collate all the small chunks. We use a while loop to read the file in small parts and track the process. We then store each chunk in the chunk array and update the progress. We combine all chunks into one final file and log Download complete! when the process is completed.

5. For multi-threading, use SharedArrayBuffer. Some applications (e.g., Web Workers) require multitasking instead of one single way of processing; it is recommended to use SharedArrayBuffer to share data between threads without copying. This is most often used for high-performance tasks like video processing or real-time communication.

Here is how to make use of SharedArrayBuffer:

const buffer = new SharedArrayBuffer(1024); // Shared across threads
const uint8Array = new Uint8Array(buffer);

Conclusion

ArrayBuffer in JavaScript helps handle binary data efficiently. Although it may seem complex to understand, it can help develop high-performance software applications when implemented correctly. In fact, it is the technology behind many successful streaming platforms and other real-time applications.