Understanding Slices in Go Programming

Aug 15, 2013


Since I started programming in Go the concept and use of slices has been confusing. This is something completely new to me. They look like an array, and feel like an array, but they are much more than an array. I am constantly reading how slices are used quite a bit by Go programmers and I think it is finally time for me to understand what slices are all about.

There is a great blog post written by Andrew Gerrand about slices:

http://blog.golang.org/go-slices-usage-and-internals

There is no reason to repeat everything Andrew has written so please read his post before continuing. Let’s just cover the internals of a slice.


The picture above represents the internal structure of a slice. When you allocate a slice this data structure along with an underlying array is created. The value of your slice variable will be this data structure. When you pass a slice to a function, a copy of this data structure is created on the stack.

We can create a slice in two ways:

Here we use the keyword make, passing the type of data we are storing, the initial length of the slice and the capacity of the underlying array.

mySlice := make([]string, 5, 8)
mySlice[0] = "Apple"
mySlice[1] = "Orange"
mySlice[2] = "Banana"
mySlice[3] = "Grape"
mySlice[4] = "Plum"

// You don’t need to include the capacity. Length and Capacity will be the same
mySlice := make([]string, 5)

You can also use a slice literal. In this case the length and capacity will be the same. Notice no value is provided inside the hard brackets []. If you add a value you will have an array. If you don’t add a value you will have a slice.

mySlice := []string{"Apple", "Orange", "Banana", "Grape", "Plum"}

You can’t extend the capacity of a slice once it is created. The only way to change the capacity is to create a new slice and perform a copy. Andrew provides a great sample function that shows an efficient way to check the remaining capacity for room and only if necessary, perform a copy.

The length of a slice identifies the number of elements of the underlying array we are using from the starting index position. The capacity identifies the number of elements we have available for use.

We can create a new slice from the original slice:

newSlice := mySlice[2:4]


The value of the new slice’s pointer variable is associated with index positions 2 and 3 of the initial underlying array. As far as this new slice is concerned, we now have a underlying array of 3 elements and we are only using 2 of those 3 elements. This new slice has no knowledge of the first two elements from the initial underlying array and never will.

When performing a slice operation the first parameter specifies the starting index from the slices pointer variable position. In our case we said index 2 which is 3 elements inside the initial underlying array we are taking the slice from. The second parameter is the last index position plus one (+1). In our case we said index 4 which will include all indexes between index 2 (the starting position) and index 3 (the final position).

We don’t always need to include a starting or ending index position when performing a slice operation:

newSlice2 = newSlice[:cap(newSlice)]


In this example we use the new slice we created before to create a third slice. We don’t provide a starting index position but we do specify the last index position. Our latest slice has the same starting position and capacity but the length has changed. By specifying the last index position as the capacity size, this length of this slice uses all remaining elements from the underlying array.

Now let’s run some code to prove this data structure actually exists and slices work as advertised.

I have created a function that will inspect the memory associated with any slice:

func InspectSlice(slice []string) {
    // Capture the address to the slice structure
    address := unsafe.Pointer(&slice)

    // Capture the address where the length and cap size is stored
    lenAddr := uintptr(address) + uintptr(8)
    capAddr := uintptr(address) + uintptr(16)

    // Create pointers to the length and cap size
    lenPtr := (*int)(unsafe.Pointer(lenAddr))
    capPtr := (*int)(unsafe.Pointer(capAddr))

    // Create a pointer to the underlying array
    addPtr := (*[8]string)(unsafe.Pointer(*(*uintptr)(address)))

    fmt.Printf("Slice Addr[%p] Len Addr[0x%x] Cap Addr[0x%x]\n",
        address,
        lenAddr,
        capAddr)

    fmt.Printf("Slice Length[%d] Cap[%d]\n",
        *lenPtr,
        *capPtr)

    for index := 0; index < *lenPtr; index++ {
        fmt.Printf("[%d] %p %s\n",
            index,
            &(*addPtr)[index],
            (*addPtr)[index])
    }

    fmt.Printf("\n\n")
}

This function is performing a bunch of pointer manipulations so we can inspect the memory and values of a slice’s data structure and underlying array.

We will break it down, but first let’s create a slice and run it through the inspect function:

package main

import (
    "fmt"
    "unsafe"
)

func main() {
    orgSlice := make([]string, 5, 8)
    orgSlice[0] = "Apple"
    orgSlice[1] = "Orange"
    orgSlice[2] = "Banana"
    orgSlice[3] = "Grape"
    orgSlice[4] = "Plum"

    InspectSlice(orgSlice)
}

Here is the output of the program:

Slice Addr[0x2101be000] Len Addr[0x2101be008] Cap Addr[0x2101be010]
Slice Length[5] Cap[8]
[0] 0x2101bd000 Apple
[1] 0x2101bd010 Orange
[2] 0x2101bd020 Banana
[3] 0x2101bd030 Grape
[4] 0x2101bd040 Plum

It appears the slice’s data structure really does exist as described by Andrew.

The InspectSlice function first displays the address of the slice’s data structure and the address positions where the length and capacity values should be. Then by creating int pointers using those addresses, we display the values for length and capacity. Last we create a pointer to the underlying array. Using the pointer, we iterate through the underlying array displaying the index position, the starting address of the element and the value.

Let’s break down the InspectSlice function to understand how it works:

// Capture the address to the slice structure
address := unsafe.Pointer(&slice)

// Capture the address where the length and cap size is stored
lenAddr := uintptr(address) + uintptr(8)
capAddr := uintptr(address) + uintptr(16)

unsafe.Pointer is a special type that is mapped to an uintptr type. Because we need to perform pointer arithmetic, we need to work with generic pointers. The first line of code casts the address of the slice’s data structure to an unsafe.Pointer. Then we create two generic pointers that point 8 and 16 bytes into the slice’s data structure respectively.

The following diagram shows each pointer variable, the value of the variable and the value that the pointer points to:

address lenAddr capAddr
0x2101be000 0x2101be008 0x2101be010
0x2101bd000 5 8

With our pointers in hand, we can now create properly typed pointers so we can display the values. Here we create two integer pointers that can be used to display the length and capacity values from the slice’s data structure.

// Create pointers to the length and cap size
lenPtr := (*int)(unsafe.Pointer(lenAddr))
capPtr := (*int)(unsafe.Pointer(capAddr))

We now need a pointer of type [8]string, which is the type of underlying array.

// Create a pointer to the underlying array
addPtr := (*[8]string)(unsafe.Pointer(*(*uintptr)(address)))

There is a lot going on in this one statement so let’s break it down:

(*uintptr)(address) : 0x2101be000
This code takes the starting address of the slice’s data structure and casts it as a generic pointer.

(*uintptr)(address) : 0x2101bd000
Then we get the value that the pointer is pointing to, which is the starting address of the underlying array.

unsafe.Pointer(*(*uintptr)(address))
Then we cast the starting address of the underlying array to an unsafe.Pointer type. We need a pointer of this type to perform the final cast.

(*[8]string)(unsafe.Pointer(*(*uintptr)(address)))
Finally we cast the unsafe.Pointer to a pointer of the proper type.

The remaining code uses the proper pointers to display the output:

fmt.Printf("Slice Addr[%p] Len Addr[0x%x] Cap Addr[0x%x]\n",
    address,
    lenAddr,
    capAddr)

fmt.Printf("Slice Length[%d] Cap[%d]\n",
    *lenPtr,
    *capPtr)

for index := 0; index < *lenPtr; index++ {
    fmt.Printf("[%d] %p %s\n",
        index,
        &(*addPtr)[index],
        (*addPtr)[index])
}

Now let’s put the entire program together and create some slices. We will inspect each slice and make sure everything we know about slices is true:

package main

import (
    "fmt"
    "unsafe"
)

func main() {
    orgSlice := make([]string, 5, 8)
    orgSlice[0] = "Apple"
    orgSlice[1] = "Orange"
    orgSlice[2] = "Banana"
    orgSlice[3] = "Grape"
    orgSlice[4] = "Plum"

    InspectSlice(orgSlice)

    slice2 := orgSlice[2:4]
    InspectSlice(slice2)

    slice3 := slice2[1:cap(slice2)]
    InspectSlice(slice3)

    slice3[0] = "CHANGED"
    InspectSlice(slice3)
    InspectSlice(slice2)
}

func InspectSlice(slice []string) {
    // Capture the address to the slice structure
    address := unsafe.Pointer(&slice)

    // Capture the address where the length and cap size is stored
    lenAddr := uintptr(address) + uintptr(8)
    capAddr := uintptr(address) + uintptr(16)

    // Create pointers to the length and cap size
    lenPtr := (*int)(unsafe.Pointer(lenAddr))
    capPtr := (*int)(unsafe.Pointer(capAddr))

    // Create a pointer to the underlying array
    addPtr := (*[8]string)(unsafe.Pointer(*(*uintptr)(address)))

    fmt.Printf("Slice Addr[%p] Len Addr[0x%x] Cap Addr[0x%x]\n",
        address,
        lenAddr,
        capAddr)

    fmt.Printf("Slice Length[%d] Cap[%d]\n",
        *lenPtr,
        *capPtr)

    for index := 0; index < *lenPtr; index++ {
        fmt.Printf("[%d] %p %s\n",
            index,
            &(*addPtr)[index],
            (*addPtr)[index])
    }

    fmt.Printf("\n\n")
}

Here is the code and output for each slice:

Here we create the initial slice with a length of 5 elements and a capacity of 8 elements.

Code:
orgSlice := make([]string, 5, 8)
orgSlice[0] = "Apple"
orgSlice[1] = "Orange"
orgSlice[2] = "Banana"
orgSlice[3] = "Grape"
orgSlice[4] = "Plum"

Output:
Slice Addr[0x2101be000] Len Addr[0x2101be008] Cap Addr[0x2101be010]
Slice Length[5] Cap[8]
[0] 0x2101bd000 Apple
[1] 0x2101bd010 Orange
[2] 0x2101bd020 Banana
[3] 0x2101bd030 Grape
[4] 0x2101bd040 Plum

The output is as expected. A length of 5, capacity of 8 and the underlying array contains our values.

Next we take a slice from the original slice. We ask for 2 elements between indexes 2 and 3.

Code:
slice2 := orgSlice[2:4]
InspectSlice(slice2)

Output:
Slice Addr[0x2101be060] Len Addr[0x2101be068] Cap Addr[0x2101be070]
Slice Length[2] Cap[6]
[0] 0x2101bd020 Banana
[1] 0x2101bd030 Grape

In the output you can see we have a slice with a length of 2 and a capacity of 6. Because this new slice is starting 3 elements into the underlying array for the original slice, there is a capacity of 6 elements. The capacity includes all possible elements that can be accessed by the new slice. Index 0 of the new slice maps to index 2 of the original slice. They both have the same address of 0x2101bd020.

This time we ask for a slice starting from index position 1 up to the last element of slice2.

Code:
slice3 := slice2[1:cap(slice2)]
InspectSlice(slice3)

Output:
Slice Addr[0x2101be0a0] Len Addr[0x2101be0a8] Cap Addr[0x2101be0b0]
Slice Length[5] Cap[5]
[0] 0x2101bd030 Grape
[1] 0x2101bd040 Plum
[2] 0x2101bd050
[3] 0x2101bd060
[4] 0x2101bd070

As expected the length and the capacity are both 5. When we display all the values of the slice you see the last three elements don’t have a value. The slice initialized all the elements when the underlying array was created. Also index 0 of this slice maps to index 1 of slice 2 and index 3 of the original slice. They all have the same address of 0x2101bd030.

The final code changes the value of the first element, index 0 in slice3 to the value CHANGED. Then we display the values for slice3 and slice2.

slice3[0] = "CHANGED"
InspectSlice(slice3)
InspectSlice(slice2)

Slice Addr[0x2101be0e0] Len Addr[0x2101be0e8] Cap Addr[0x2101be0f0]
Slice Length[5] Cap[5]
[0] 0x2101bd030 CHANGED
[1] 0x2101bd040 Plum
[2] 0x2101bd050
[3] 0x2101bd060
[4] 0x2101bd070


Slice Addr[0x2101be120] Len Addr[0x2101be128] Cap Addr[0x2101be130]
Slice Length[2] Cap[6]
[0] 0x2101bd020 Banana
[1] 0x2101bd030 CHANGED

Notice that both slices show the changed value in their respect indexes. This proves all the slices are using the same underlying array.

The InspectSlice function proves that each slice contains its own data structure with a pointer to an underlying array, a length for the slice and a capacity. Take some time to create more slices and use the InspectSlice function to validate your assumptions.



Go Training

We have taught Go to thousands of developers all around the world since 2014. There is no other company that has been doing it longer and our material has proven to help jump start developers 6 to 12 months ahead of their knowledge of Go. We know what knowledge developers need in order to be productive and efficient when writing software in Go.

Our Go, Web and Data Science classes are perfect for both experienced and beginning engineers. We start every class from the beginning and get very detailed about the internals, mechanics, specification, guidelines, best practices and design philosophies. We cover a lot about "if performance matters" with a focus on mechanical sympathy, data oriented design, decoupling and writing production software.

Learn More

To learn about Corporate training events, options and special pricing please contact:

William Kennedy
ArdanLabs (www.ardanlabs.com)
bill@ardanlabs.com