Three-Index Slices in Go 1.2

Dec 17, 2013


With the release of Go 1.2, slices gained the ability to specify the capacity when performing a slicing operation. This doesn’t mean we can use this index to extend the capacity of the underlying array. It means we can create a new slice whose capacity is restricted. Restricting the capacity provides a level of protection to the underlying array and gives us more control over append operations.

Here are the release notes and design document for the feature request:

http://tip.golang.org/doc/go1.2#three_index
https://docs.google.com/document/d/1GKKdiGYAghXRxC2BFrSEbHBZZgAGKQ-yXK-hRKBo0Kk/pub

Let’s write some code to explore using the new capacity index. As with all my slice posts, I am going to use this InspectSlice function:

func InspectSlice(slice []string) {
    // Capture the address to the slice structure
    address := unsafe.Pointer(&slice)

    // Capture the address where the length and cap size is stored
    lenAddr := uintptr(address) + uintptr(8)
    capAddr := uintptr(address) + uintptr(16)

    // Create pointers to the length and cap size
    lenPtr := (*int)(unsafe.Pointer(lenAddr))
    capPtr := (*int)(unsafe.Pointer(capAddr))

    // Create a pointer to the underlying array
    addPtr := (*[8]string)(unsafe.Pointer(*(*uintptr)(address)))

    fmt.Printf("Slice Addr[%p] Len Addr[0x%x] Cap Addr[0x%x]\n",
        address,
        lenAddr,
        capAddr)

    fmt.Printf("Slice Length[%d] Cap[%d]\n",
        *lenPtr,
        *capPtr)

    for index := 0; index < *lenPtr; index++ {
        fmt.Printf("[%d] %p %s\n",
            index,
            &(*addPtr)[index],
            (*addPtr)[index])
    }

    fmt.Printf("\n\n")
}

To start, let’s create a slice we will use as our source:

source := []string{"Apple", "Orange", "Plum", "Banana", "Grape"}
InspectSlice(source)


Output:
Slice Addr[0x210231000] Len Addr[0x210231008] Cap Addr[0x210231010]
Slice Length[5] Cap[5]
[0] 0x21020e140 Apple
[1] 0x21020e150 Orange
[2] 0x21020e160 Plum
[3] 0x21020e170 Banana
[4] 0x21020e180 Grape

We start with a slice of strings with a length and capacity of 5. This means the underlying array has 5 elements and we have access to the entire array.

Next, let’s take a traditional slice of the source and inspect the contents:

takeOne := source[2:3]
InspectSlice(takeOne)


Output:
Slice Addr[0x210231040] Len Addr[0x210231048] Cap Addr[0x210231050]
Slice Length[1] Cap[3]
[0] 0x21020e160 Plum

With this slice operation we only take the third element from the source. You can see the first element of the takeOne slice has the same address as the third element of the source slice. The takeOne slice has a length of one and a capacity of three. This is because there are three elements left in the underlying array that are available for use.

What if we didn’t want the new slice to have access to the remaining capacity? Prior to version 1.2, this was not possible. Let’s take the slice again, but this time restrict the capacity to one:

takeOneCapOne := source[2:3:3]  // Use the third index position to
InspectSlice(takeOneCapOne)     // set the capacity


Output:
Slice Addr[0x210231060] Len Addr[0x210231068] Cap Addr[0x210231070]
Slice Length[1] Cap[1]
[0] 0x21020e160 Plum

After creating the takeOneCapOne slice, the length and capacity are now one. The takeOneCapOne slice no longer has access to the remaining capacity in the underlying array.

Length and capacity is calculated using this formula:

For slice[ i : j : k ] the

Length:   j - i
Capacity: k - i

If we attempt to set the capacity greater than the underlying array, the code will panic.

takeOneCapFour := source[2:3:6]  // (6 - 2) attempts to set the capacity
                                 // to 4. This is greater than what is
                                 // available.

Runtime Error:
panic: runtime error: slice bounds out of range

goroutine 1 [running]:
runtime.panic(0x9ad20, 0x1649ea)
    /Users/bill/go/src/pkg/runtime/panic.c:266 +0xb6
main.main()
    /Users/bill/Spaces/Test/src/test/main.go:15 +0x24f

So what happens if we append an element to the takeOneCapOne slice?

source := []string{"Apple", "Orange", "Plum", "Banana", "Grape"}
InspectSlice(source)

takeOneCapOne := source[2:3:3]
InspectSlice(takeOneCapOne)

takeOneCapOne = append(takeOneCapOne, "Kiwi")
InspectSlice(takeOneCapOne)

Here is the output:

Slice Addr[0x210231000] Len Addr[0x210231008] Cap Addr[0x210231010]
Slice Length[5] Cap[5]
[0] 0x21020e140 Apple
[1] 0x21020e150 Orange
[2] 0x21020e160 Plum
[3] 0x21020e170 Banana
[4] 0x21020e180 Grape

– Before Append –

Slice Addr[0x210231040] Len Addr[0x210231048] Cap Addr[0x210231050]
Slice Length[1] Cap[1]
[0] 0x21020e160 Plum

– After Append –

Slice Addr[0x210231080] Len Addr[0x210231088] Cap Addr[0x210231090]
Slice Length[2] Cap[2]
[0] 0x210231060 Plum
[1] 0x210231070 Kiwi

When we append an element to the takeOneCapOne slice, a new underlying array is created for the slice. This new underlying array contains a copy of the elements being referenced from the source and then is extended to add the new element. This is because the capacity of the takeOneCapOne slice was reached and append needed to grow the capacity. Notice how the address changes in the takeOneCapOne slice after the append.

How is this different from not setting the capacity?

source := []string{"Apple", "Orange", "Plum", "Banana", "Grape"}
InspectSlice(source)

takeOne := source[2:3]  // Don’t specify capacity
InspectSlice(takeOne)

takeOne = append(takeOne, "Kiwi")
InspectSlice(takeOne)

InspectSlice(source)

Here is the output:

Slice Addr[0x210231000] Len Addr[0x210231008] Cap Addr[0x210231010]
Slice Length[5] Cap[5]
[0] 0x21020e140 Apple
[1] 0x21020e150 Orange
[2] 0x21020e160 Plum
[3] 0x21020e170 Banana
[4] 0x21020e180 Grape

– Before Append –

Slice Addr[0x210231040] Len Addr[0x210231048] Cap Addr[0x210231050]
Slice Length[1] Cap[3]
[0] 0x21020e160 Plum

– After Append –

Slice Addr[0x210231060] Len Addr[0x210231068] Cap Addr[0x210231070]
Slice Length[2] Cap[3]
[0] 0x21020e160 Plum
[1] 0x21020e170 Kiwi


Slice Addr[0x210231080] Len Addr[0x210231088] Cap Addr[0x210231090]
Slice Length[5] Cap[5]
[0] 0x21020e140 Apple
[1] 0x21020e150 Orange
[2] 0x21020e160 Plum
[3] 0x21020e170 Kiwi
[4] 0x21020e180 Grape

This time the append uses the existing capacity and overwrites the value at element 4 in the underlying array. This could be a disaster if this was not our intent.

This new feature of setting the capacity can really help protect us and our data from unwanted overwrites. The more we can leverage the built-in functions and runtime to handle these types of operations the better. These types of bugs are very difficult to find so this is going to help immeasurably.

Here are other posts about slices:

Understanding Slices In Go Programming
Collections Of Unknown Length In Go
Slices Of Slices Of Slices In Go
Iterating Over Slices In Go


Ultimate Go Programming LiveLessons

Ultimate Go Programming LiveLessons provides an intensive, comprehensive, and idiomatic view of the Go programming language. This course focuses on both the specification and implementation of the language, including topics ranging from language syntax, design, and guidelines to concurrency, testing, and profiling. This class is perfect for anyone who wants a jump-start in learning Go or wants a more thorough understanding of the language and its internals.

Learn more

Go Training

We have taught Go to thousands of developers all around the world since 2014. There is no other company that has been doing it longer and our material has proven to help jump start developers 6 to 12 months ahead of their knowledge of Go. We know what knowledge developers need in order to be productive and efficient when writing software in Go.

Our Go, Web and Data Science classes are perfect for both experienced and beginning engineers. We start every class from the beginning and get very detailed about the internals, mechanics, specification, guidelines, best practices and design philosophies. We cover a lot about "if performance matters" with a focus on mechanical sympathy, data oriented design, decoupling and writing production software.

Learn More

To learn about Corporate training events, options and special pricing please contact:

William Kennedy
ArdanLabs (www.ardanlabs.com)
bill@ardanlabs.com