rockhopper 0.2.0

Higher Dimensional Arrays

Rockhopper is very much geared towards 2D ragged arrays, however, one permutation of higher dimensional ragged arrays is allowed: A ragged array’s rows can be multidimensional rather than a 1D arrays.

Construction works more or less as you’d expect. The following shows 3 different ways to create the same multidimensional ragged array.

import numpy as np
from rockhopper import ragged_array, RaggedArray

# Construct from nested lists.
from_nested = ragged_array([
    [[0,  1], [2, 3]],
    [[4, 5]],
    [[6, 7], [8, 9], [10, 11]],
    [[12, 13]],
])

# Construction from flat contents and either ...
flat = np.array([
    [0,  1], [2, 3], [4, 5], [6, 7], [8, 9], [10, 11], [12, 13]
])
# ... row lengths, ...
from_lengths = RaggedArray.from_lengths(flat, [2, 1, 3, 2])
# ... or row bounds.
from_bounds = RaggedArray(flat, [0, 2, 3, 6, 7])

Structured Arrays

Ragged arrays may also use a structured data type. For this, explicitly setting the dtype parameter is mandatory when using the ragged_array() constructor. Otherwise NumPy will cast everything to one compatible type (usually str).

ragged = ragged_array([
    [("abc", 3), ("efg", 5)],
    [("hij", 1)],
    [("klm", 13), ("nop", 99), ("qrs", 32)],
], dtype=[("foo", str, 3), ("bar", int)])

However, this feature is only half-formed because ragged["foo"] requires internal support for strided flat arrays (which rockhopper currently lacks).

1D indices

Indexing will all be shown by examples. Here is an unimaginative ragged array to play with.

from rockhopper import ragged_array

ragged = ragged_array([
    [1, 2, 3, 4],
    [5, 6],
    [7, 8, 9],
    [10, 11, 12, 13],
])

1D indexing with individual integers gives single rows as regular arrays.

>>> ragged[2]
array([7, 8, 9])
>>> ragged[3]
array([10, 11, 12, 13])

But indexing with a slice, integer array or bool mask gives another ragged array.

>>> ragged[::2]
RaggedArray.from_nested([
    [1, 2, 3, 4],
    [7, 8, 9],
])
>>> ragged[[2, -1]]
RaggedArray.from_nested([
    [7, 8, 9],
    [10, 11, 12, 13],
])

This is true even if all rows happen to be the same length.

2D indices

2D indexing ragged[rows, columns] gives individual cells. Arrays of indices, slices and bool masks may also be used instead of single numbers. Using the same boring ragged array as above:

# Individual indices.
>>> ragged[0, 0], ragged[0, 1], ragged[0, 2]
(1, 2, 3)

# Arrays of indices.
>>> ragged[0, [0, 1, -1]]
array([1, 2, 4])
>>> ragged[0, [[1, 2], [0, 2]]]
array([[2, 3],
       [1, 3]])
>>> ragged[[0, 3, 2], [2, 3, 1]]
array([ 3, 13,  8])

# Slices as row numbers (including the null slice [:]).
>>> ragged[:, 0]
array([ 1,  5,  7, 10])
>>> ragged[2:, -1]
array([ 9, 13])

# Again, multiple column numbers may be given.
# The following gets the first and last element from each row.
>>> ragged[:, [0, -1]]
array([[ 1,  4],
       [ 5,  6],
       [ 7,  9],
       [10, 13]])

# If the second index is a slice or bool mask, the output is a ragged array.
# Even if each row is of the same length.
>>> ragged[:, :2]
RaggedArray.from_nested([
    [1, 2],
    [5, 6],
    [7, 8],
    [10, 11],
])

If the second index is not a slice then the the output of getitem is a copy and does not share memory with the parent ragged array.

3D (or higher) indices

Higher Dimensional Arrays can be sliced using 3 indices (or more).

Using another uninspiring enumeration example - this time a 3D array:

ragged = ragged_array([
    [[ 0,  1,  2], [ 3,  4,  5]],
    [[ 6,  7,  8], [ 9, 10, 11]],
    [[12, 13, 14], [15, 16, 17], [18, 19, 20]],
    [[21, 22, 23]],
])

3D arrays follow the same indexing rules as 2D arrays except that each cell is actually another array.

>>> ragged[0, 1]
array([3, 4, 5])

And a triplet of indices are used to access individual elements.

>>> ragged[2, 0, 1]
13

To list of lists

The tolist() method converts back to nested lists (like those used to build the array in the first place).

>>> ragged.tolist()
[[1, 2, 3], [4, 5, 6], [7, 8], [9, 10], [11, 12, 13]]

To list of homogenous arrays

When a ragged array is either not very ragged (row lengths are mostly the same) or not ragged at all (rows are all the same length), it’s often helpful to split it on rows of differing lengths, giving a sequence of standard rectangular arrays which can be for loop-ed over. Do this with the to_rectangular_arrays() method.

>>> ragged.to_rectangular_arrays()
[array([[1, 2, 3],
        [4, 5, 6]]),
 array([[7, 8],
        [9, 10]]),
 array([[11, 12, 13]])]

In the somewhat unlikely event that you don’t care about the order the rows appear in, set the reorder option to allow it to presort the rows into ascending lengths so as to minimize fragmentation.

>>> sort_args, arrays = ragged.to_rectangular_arrays(reorder=True)
# The numpy.argsort() arguments are returned in case you want them.
>>> sort_args
array([2, 3, 0, 1, 4])
# By sorting, only 2 arrays are needed rather than 3.
>>> arrays
[array([[ 7,  8],
        [ 9, 10]]),
 array([[ 1,  2,  3],
        [ 4,  5,  6],
        [11, 12, 13]])]

Binary

Currently rockhopper knows of exactly one binary format: The highly typical, but hopelessly un-NumPy-friendly:

row-length | row-content | row-length | row-content

binary form often found in 3D graphics where row-length may be any unsigned integer type of either byteorder, row-content may be of any data type or byteorder, and there are no delimiters or metadata anywhere.

For this format RaggedArray() provides a loads() method for reading and a dumps() method for writing.

Some examples:

# Write using:
#  - Row contents: The current data type (ragged.dtype) and endian.
#  - Row lengths: ``numpy.intc`` native endian
# Note that the output is a memoryview() which is generally interchangeable
# with bytes(). This may still be written to a file with the usual
# ``fh.write()``.
dumped = ragged.dumps()

# Read back using:
#  - Row contents: The same dtype used to write it
#  - Row lengths: ``numpy.intc`` native endian
ragged, bytes_consumed = RaggedArray.loads(dumped, ragged.dtype)

# Write then read using:
#  - Row contents: Big endian 8-byte floats
#  - Row lengths: Little endian 2-byte unsigned integers
dumped = ragged.astype(">f8").dumps(ldtype="<u2")
ragged, bytes_consumed = RaggedArray.loads(dumped, ">f8", ldtype="<u2")

By default, loads() will keep adding rows until it hits the end of the byte array that it’s parsing. The bytes_consumed (a count of how many bytes from dumped where used) will therefore always satisfy bytes_consumed == len(dumped).

Some file formats contain a serialised ragged array embedded inside a larger file but don’t specify how many bytes belong to the ragged array and how many belong to whatever comes afterwards. Instead they specify how many rows there should be. To read such data use the rows keyword argument.

# Read a 20 row ragged array of floats from a long ``bytes()`` object called
# **blob**. Will raise an error if it runs out of data.
ragged, bytes_consumed = ragged.loads(blob, "f8", rows=20)

# ``bytes_consumed`` indicates where the ragged array stopped.
rest_of_blob = blob[bytes_consumed:]

Pickle

If you don’t need other programs to be able to read the output then bog-standard pickle works too.

>>> import pickle
>>> arr = ragged_array([
...    ["cake", "biscuits"],
...    ["socks"],
...    ["orange", "lemon", "pineapple"],
... ])
>>> pickle.loads(pickle.dumps(arr))
RaggedArray.from_nested([
    ["cake", "biscuits"],
    ["socks"],
    ["orange", "lemon", "pineapple"],
])

Enumerating classes

The above assumes that the parameter you wish to group by is just an enumeration. If this is not the case, and you’re not already sick of software written by me, then you may use a hirola.HashTable() to efficiently enumerate the parameter to group by.

For example, to group this list of animals by their animal class:

animals = np.array([
    ("cow", "mammal"),
    ("moose", "mammal"),
    ("centipede", "insect"),
    ("robin", "bird"),
    ("spider", "insect"),
    ("whale", "mammal"),
    ("woodpecker", "bird"),
], dtype=[("name", str, 15), ("class", str, 15)])

Use something like:

>>> from hirola import HashTable
>>> animal_classes = HashTable(len(animals), animals.dtype["class"])
>>> enum = animal_classes.add(animals["class"])

>>> RaggedArray.group_by(animals["name"], enum)
RaggedArray.from_nested([
    ['cow', 'moose', 'whale'],
    ['centipede', 'spider'],
    ['robin', 'woodpecker'],
])
>>> animal_classes.keys
array(['mammal', 'insect', 'bird'], dtype='<U15')