navigate.model.concurrency.concurrency_tools.SharedNDArray
- class navigate.model.concurrency.concurrency_tools.SharedNDArray(shape=None, dtype=<class 'float'>, shared_memory_name=None, offset=0, strides=None, order=None)
Bases:
ndarrayA numpy array that lives in shared memory
Inputs and outputs to/from ObjectInSubprocess are ‘serialized’, which is pretty fast - except for large in-memory objects. The only large in-memory objects we regularly deal with are numpy arrays, so it makes sense to provide a way to pass large numpy arrays via shared memory (which avoids slow serialization).
Maybe you wanted to write code that looks like this:
data_buf = np.zeros((400, 2000, 2000), dtype=’uint16’) display_buf = np.zeros((2000, 2000), dtype=’uint8’)
camera = Camera() preprocessor = Preprocessor() display = Display()
camera.record(num_images=400, out=data_buf) preprocessor.process(in=data_buf, out=display_buf) display.show(display_buf)
…but instead you write code that looks like this:
data_buf = SharedNDArray(shape=(400, 2000, 2000), dtype=’uint16’) display_buf = SharedNDArray(shape=(2000, 2000), dtype=’uint8’)
camera = ObjectInSubprocess(Camera) preprocessor = ObjectInSubprocess(Preprocessor) display = ObjectInSubprocess(Display)
camera.record(num_images=400, out=data_buf) preprocessor.process(in=data_buf, out=display_buf) display.show(display_buf)
…and your payoff is, each object gets its own CPU core, AND passing large numpy arrays between the processes is still really fast!
To implement this we used memmap from numpy.core as a template.
Attributes
- T
View of the transposed array.
Same as
self.transpose().Examples
>>> import numpy as np >>> a = np.array([[1, 2], [3, 4]]) >>> a array([[1, 2], [3, 4]]) >>> a.T array([[1, 3], [2, 4]])
>>> a = np.array([1, 2, 3, 4]) >>> a array([1, 2, 3, 4]) >>> a.T array([1, 2, 3, 4])
See also
transpose
- base
Base object if memory is from some other object.
Examples
The base of an array that owns its memory is None:
>>> import numpy as np >>> x = np.array([1,2,3,4]) >>> x.base is None True
Slicing creates a view, whose memory is shared with x:
>>> y = x[2:] >>> y.base is x True
- ctypes
An object to simplify the interaction of the array with the ctypes module.
This attribute creates an object that makes it easier to use arrays when calling shared libraries with the ctypes module. The returned object has, among others, data, shape, and strides attributes (see Notes below) which themselves return ctypes objects that can be used as arguments to a shared library.
- Parameters:
None –
- Returns:
c – Possessing attributes data, shape, strides, etc.
- Return type:
Python object
See also
Notes
Below are the public attributes of this object which were documented in “Guide to NumPy” (we have omitted undocumented public attributes, as well as documented private attributes):
- _ctypes.data
A pointer to the memory area of the array as a Python integer. This memory area may contain data that is not aligned, or not in correct byte-order. The memory area may not even be writeable. The array flags and data-type of this array should be respected when passing this attribute to arbitrary C-code to avoid trouble that can include Python crashing. User Beware! The value of this attribute is exactly the same as:
self._array_interface_['data'][0].Note that unlike
data_as, a reference won’t be kept to the array: code likectypes.c_void_p((a + b).ctypes.data)will result in a pointer to a deallocated array, and should be spelt(a + b).ctypes.data_as(ctypes.c_void_p)
- _ctypes.shape
A ctypes array of length self.ndim where the basetype is the C-integer corresponding to
dtype('p')on this platform (see ~numpy.ctypeslib.c_intp). This base-type could be ctypes.c_int, ctypes.c_long, or ctypes.c_longlong depending on the platform. The ctypes array contains the shape of the underlying array.- Type:
(c_intp*self.ndim)
- _ctypes.strides
A ctypes array of length self.ndim where the basetype is the same as for the shape attribute. This ctypes array contains the strides information from the underlying array. This strides information is important for showing how many bytes must be jumped to get to the next element in the array.
- Type:
(c_intp*self.ndim)
- _ctypes.data_as(obj)
Return the data pointer cast to a particular c-types object. For example, calling
self._as_parameter_is equivalent toself.data_as(ctypes.c_void_p). Perhaps you want to use the data as a pointer to a ctypes array of floating-point data:self.data_as(ctypes.POINTER(ctypes.c_double)).The returned pointer will keep a reference to the array.
- _ctypes.shape_as(obj)
Return the shape tuple as an array of some other c-types type. For example:
self.shape_as(ctypes.c_short).
- _ctypes.strides_as(obj)
Return the strides tuple as an array of some other c-types type. For example:
self.strides_as(ctypes.c_longlong).
If the ctypes module is not available, then the ctypes attribute of array objects still returns something useful, but ctypes objects are not returned and errors may be raised instead. In particular, the object will still have the
as_parameterattribute which will return an integer equal to the data attribute.Examples
>>> import numpy as np >>> import ctypes >>> x = np.array([[0, 1], [2, 3]], dtype=np.int32) >>> x array([[0, 1], [2, 3]], dtype=int32) >>> x.ctypes.data 31962608 # may vary >>> x.ctypes.data_as(ctypes.POINTER(ctypes.c_uint32)) <__main__.LP_c_uint object at 0x7ff2fc1fc200> # may vary >>> x.ctypes.data_as(ctypes.POINTER(ctypes.c_uint32)).contents c_uint(0) >>> x.ctypes.data_as(ctypes.POINTER(ctypes.c_uint64)).contents c_ulong(4294967296) >>> x.ctypes.shape <numpy._core._internal.c_long_Array_2 object at 0x7ff2fc1fce60> # may vary >>> x.ctypes.strides <numpy._core._internal.c_long_Array_2 object at 0x7ff2fc1ff320> # may vary
- data
Python buffer object pointing to the start of the array’s data.
- device
- dtype
Data-type of the array’s elements.
Warning
Setting
arr.dtypeis discouraged and may be deprecated in the future. Setting will replace thedtypewithout modifying the memory (see also ndarray.view and ndarray.astype).- Parameters:
None –
- Returns:
d
- Return type:
numpy dtype object
See also
ndarray.astypeCast the values contained in the array to a new data-type.
ndarray.viewCreate a view of the same data but a different data-type.
Examples
>>> import numpy as np >>> x = np.arange(4).reshape((2, 2)) >>> x array([[0, 1], [2, 3]]) >>> x.dtype dtype('int64') # may vary (OS, bitness) >>> isinstance(x.dtype, np.dtype) True
- flags
Information about the memory layout of the array.
- C_CONTIGUOUS(C)
The data is in a single, C-style contiguous segment.
- F_CONTIGUOUS(F)
The data is in a single, Fortran-style contiguous segment.
- OWNDATA(O)
The array owns the memory it uses or borrows it from another object.
- WRITEABLE(W)
The data area can be written to. Setting this to False locks the data, making it read-only. A view (slice, etc.) inherits WRITEABLE from its base array at creation time, but a view of a writeable array may be subsequently locked while the base array remains writeable. (The opposite is not true, in that a view of a locked array may not be made writeable. However, currently, locking a base object does not lock any views that already reference it, so under that circumstance it is possible to alter the contents of a locked array via a previously created writeable view onto it.) Attempting to change a non-writeable array raises a RuntimeError exception.
- ALIGNED(A)
The data and all elements are aligned appropriately for the hardware.
- WRITEBACKIFCOPY(X)
This array is a copy of some other array. The C-API function PyArray_ResolveWritebackIfCopy must be called before deallocating to the base array will be updated with the contents of this array.
- FNC
F_CONTIGUOUS and not C_CONTIGUOUS.
- FORC
F_CONTIGUOUS or C_CONTIGUOUS (one-segment test).
- BEHAVED(B)
ALIGNED and WRITEABLE.
- CARRAY(CA)
BEHAVED and C_CONTIGUOUS.
- FARRAY(FA)
BEHAVED and F_CONTIGUOUS and not C_CONTIGUOUS.
Notes
The flags object can be accessed dictionary-like (as in
a.flags['WRITEABLE']), or by using lowercased attribute names (as ina.flags.writeable). Short flag names are only supported in dictionary access.Only the WRITEBACKIFCOPY, WRITEABLE, and ALIGNED flags can be changed by the user, via direct assignment to the attribute or dictionary entry, or by calling ndarray.setflags.
The array flags cannot be set arbitrarily:
WRITEBACKIFCOPY can only be set
False.ALIGNED can only be set
Trueif the data is truly aligned.WRITEABLE can only be set
Trueif the array owns its own memory or the ultimate owner of the memory exposes a writeable buffer interface or is a string.
Arrays can be both C-style and Fortran-style contiguous simultaneously. This is clear for 1-dimensional arrays, but can also be true for higher dimensional arrays.
Even for contiguous arrays a stride for a given dimension
arr.strides[dim]may be arbitrary ifarr.shape[dim] == 1or the array has no elements. It does not generally hold thatself.strides[-1] == self.itemsizefor C-style contiguous arrays orself.strides[0] == self.itemsizefor Fortran-style contiguous arrays is true.
- flat
A 1-D iterator over the array.
This is a numpy.flatiter instance, which acts similarly to, but is not a subclass of, Python’s built-in iterator object.
See also
flattenReturn a copy of the array collapsed into one dimension.
flatiterExamples
>>> import numpy as np >>> x = np.arange(1, 7).reshape(2, 3) >>> x array([[1, 2, 3], [4, 5, 6]]) >>> x.flat[3] 4 >>> x.T array([[1, 4], [2, 5], [3, 6]]) >>> x.T.flat[3] 5 >>> type(x.flat) <class 'numpy.flatiter'>
An assignment example:
>>> x.flat = 3; x array([[3, 3, 3], [3, 3, 3]]) >>> x.flat[[1,4]] = 1; x array([[3, 1, 3], [3, 1, 3]])
- imag
The imaginary part of the array.
Examples
>>> import numpy as np >>> x = np.sqrt([1+0j, 0+1j]) >>> x.imag array([ 0. , 0.70710678]) >>> x.imag.dtype dtype('float64')
- itemsize
Length of one array element in bytes.
Examples
>>> import numpy as np >>> x = np.array([1,2,3], dtype=np.float64) >>> x.itemsize 8 >>> x = np.array([1,2,3], dtype=np.complex128) >>> x.itemsize 16
- mT
View of the matrix transposed array.
The matrix transpose is the transpose of the last two dimensions, even if the array is of higher dimension.
New in version 2.0.
- Raises:
ValueError – If the array is of dimension less than 2.
Examples
>>> import numpy as np >>> a = np.array([[1, 2], [3, 4]]) >>> a array([[1, 2], [3, 4]]) >>> a.mT array([[1, 3], [2, 4]])
>>> a = np.arange(8).reshape((2, 2, 2)) >>> a array([[[0, 1], [2, 3]], [[4, 5], [6, 7]]]) >>> a.mT array([[[0, 2], [1, 3]], [[4, 6], [5, 7]]])
- nbytes
Total bytes consumed by the elements of the array.
Notes
Does not include memory consumed by non-element attributes of the array object.
See also
sys.getsizeofMemory consumed by the object itself without parents in case view. This does include memory consumed by non-element attributes.
Examples
>>> import numpy as np >>> x = np.zeros((3,5,2), dtype=np.complex128) >>> x.nbytes 480 >>> np.prod(x.shape) * x.itemsize 480
- ndim
Number of array dimensions.
Examples
>>> import numpy as np >>> x = np.array([1, 2, 3]) >>> x.ndim 1 >>> y = np.zeros((2, 3, 4)) >>> y.ndim 3
- real
The real part of the array.
Examples
>>> import numpy as np >>> x = np.sqrt([1+0j, 0+1j]) >>> x.real array([ 1. , 0.70710678]) >>> x.real.dtype dtype('float64')
See also
numpy.realequivalent function
- shape
Tuple of array dimensions.
The shape property is usually used to get the current shape of an array, but may also be used to reshape the array in-place by assigning a tuple of array dimensions to it. As with numpy.reshape, one of the new shape dimensions can be -1, in which case its value is inferred from the size of the array and the remaining dimensions. Reshaping an array in-place will fail if a copy is required.
Warning
Setting
arr.shapeis discouraged and may be deprecated in the future. Using ndarray.reshape is the preferred approach.Examples
>>> import numpy as np >>> x = np.array([1, 2, 3, 4]) >>> x.shape (4,) >>> y = np.zeros((2, 3, 4)) >>> y.shape (2, 3, 4) >>> y.shape = (3, 8) >>> y array([[ 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0.], [ 0., 0., 0., 0., 0., 0., 0., 0.]]) >>> y.shape = (3, 6) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: cannot reshape array of size 24 into shape (3,6) >>> np.zeros((4,2))[::2].shape = (-1,) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: Incompatible shape for in-place modification. Use `.reshape()` to make a copy with the desired shape.
See also
numpy.shapeEquivalent getter function.
numpy.reshapeFunction similar to setting
shape.ndarray.reshapeMethod similar to setting
shape.
- size
Number of elements in the array.
Equal to
np.prod(a.shape), i.e., the product of the array’s dimensions.Notes
a.size returns a standard arbitrary precision Python integer. This may not be the case with other methods of obtaining the same value (like the suggested
np.prod(a.shape), which returns an instance ofnp.int_), and may be relevant if the value is used further in calculations that may overflow a fixed size integer type.Examples
>>> import numpy as np >>> x = np.zeros((3, 5, 2), dtype=np.complex128) >>> x.size 30 >>> np.prod(x.shape) 30
- strides
Tuple of bytes to step in each dimension when traversing an array.
The byte offset of element
(i[0], i[1], ..., i[n])in an array a is:offset = sum(np.array(i) * a.strides)
A more detailed explanation of strides can be found in The N-dimensional array (ndarray).
Warning
Setting
arr.stridesis discouraged and may be deprecated in the future. numpy.lib.stride_tricks.as_strided should be preferred to create a new view of the same data in a safer way.Notes
Imagine an array of 32-bit integers (each 4 bytes):
x = np.array([[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]], dtype=np.int32)
This array is stored in memory as 40 bytes, one after the other (known as a contiguous block of memory). The strides of an array tell us how many bytes we have to skip in memory to move to the next position along a certain axis. For example, we have to skip 4 bytes (1 value) to move to the next column, but 20 bytes (5 values) to get to the same position in the next row. As such, the strides for the array x will be
(20, 4).See also
Examples
>>> import numpy as np >>> y = np.reshape(np.arange(2 * 3 * 4, dtype=np.int32), (2, 3, 4)) >>> y array([[[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]], [[12, 13, 14, 15], [16, 17, 18, 19], [20, 21, 22, 23]]], dtype=np.int32) >>> y.strides (48, 16, 4) >>> y[1, 1, 1] np.int32(17) >>> offset = sum(y.strides * np.array((1, 1, 1))) >>> offset // y.itemsize np.int64(17)
>>> x = np.reshape(np.arange(5*6*7*8, dtype=np.int32), (5, 6, 7, 8)) >>> x = x.transpose(2, 3, 1, 0) >>> x.strides (32, 4, 224, 1344) >>> i = np.array([3, 5, 2, 2], dtype=np.int32) >>> offset = sum(i * x.strides) >>> x[3, 5, 2, 2] np.int32(813) >>> offset // x.itemsize np.int64(813)