Autograd

Python API

class singa.autograd.Abs(name=None)

y = abs(x), is applied to the tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(a)

Return abs(a), where a is CTensor.

class singa.autograd.Acos

Calculates the arccosine (inverse of cosine) of the given input tensor, element-wise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Acosh

Calculates the hyperbolic arccosine of the given input tensor element-wise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Add

Performs element-wise binary addition.

backward(dy)
Parameters

dy (CTensor) – dL / dy

Returns

a tuple for (dx0, dx1), dx0 is data for dL / da, dx1 is data for dL / db.

forward(a, b)

Return a+b, where a and b are CTensor.

class singa.autograd.AddBias(axis=0)

Add Bias to each row / column of the Tensor, depending on the axis arg.

backward(dy)
Parameters

dy (CTensor) – data for the dL / dy, L is the loss.

Returns

a tuple for (db, dx), db is data for dL / db, dx is data for dL / dx.

forward(x, b)
Parameters
  • x (CTensor) – matrix.

  • b (CTensor) – bias to be added.

Returns

the result Tensor

class singa.autograd.And

Returns the tensor resulted from performing the and logical operation elementwise on the input tensors A and B (with Numpy-style broadcasting support).

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Raises

AssertionError – no backward function for this operator

forward(a, b)

Return np.logical_and(a,b), where a and b are CTensor.

class singa.autograd.Asin

Calculates the arcsine (inverse of sine) of the given input tensor, element-wise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Asinh

Calculates the hyperbolic arcsine of the given input tensor element-wise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Atan

Calculates the arctangent (inverse of tangent) of the given input tensor, element-wise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Atanh

Calculates the hyperbolic arctangent of the given input tensor element-wise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.AvgPool1d(kernel_size, stride=None, padding=0, odd_padding=0, 0, 0, 0)

Generate a Avg Pooling 1d operator

class singa.autograd.AvgPool2d(kernel_size, stride=None, padding=0, odd_padding=0, 0, 0, 0)
class singa.autograd.BatchNorm2d(num_features, momentum=0.9)

Generate a BatchNorm 2d operator

class singa.autograd.Cast(to)

The operator casts the elements of a given input tensor to a data type specified by the ‘to’ argument and returns an output tensor of the same size in the converted type.

backward(dy)

backward of Cast Args:f

dy (CTensor), gradient tensor.

Raises

AssertionError – no backward function for this operator

forward(x)

forward of Cast :param x: input tensor. :type x: CTensor

Returns

the output CTensor.

class singa.autograd.Ceil

Ceil takes one input data (Tensor) and produces one output data (Tensor) where the ceil is, y = ceil(x), is applied to the tensor elementwise.

backward(dy)

backward of Ceil :param dy: gradient tensor. :type dy: CTensor

Returns

the gradient tensor over input tensor.

forward(x)

forward of Ceil :param x: input tensor. :type x: CTensor

Returns

the output CTensor.

class singa.autograd.Clip(min, max)

Clip operator limits the given input within an interval. The interval is specified by the inputs ‘min’ and ‘max’.

backward(dy)
Parameters

dy (CTensor) – dL / dy

Returns

dL / dx

Return type

dx (CTensor)

forward(x)
Parameters

x (CTensor) – input tensor

Returns

a new CTensor with np.clip(x,min,max)

class singa.autograd.Concat(axis=0)

Concatenate a list of tensors into a single tensor. All input tensors must have the same shape, except for the dimension size of the axis to concatenate on.

backward(dy)
Parameters

dy (CTensor) – data for the dL / dy, L is the loss

Returns

data for the dL / dxs, L is the loss,

Return type

dxs (a tuple of CTensor)

forward(*xs)
Parameters

xs (a list of CTensor) – List of tensors for concatenation

Returns

a CTensor for the result

class singa.autograd.ConstantOfShape(value=0.0)

Init a ConstantOfShape, generate a tensor with given value and shape.

backward(dy)

backward of ConstantOfShape :param dy: gradient tensor. :type dy: CTensor

Raises

AssertionError – no backward function for this operator

forward(x)

forward of ConstantOfShape :param x: CTensor, 1D tensor. The shape of the expected output tensor.

All values must be >= 0.

Returns

the output CTensor. If attribute ‘value’ is specified, the value

and datatype of the output tensor is taken from ‘value’. If attribute ‘value’ is not specified, the value in the output defaults to 0, and the datatype defaults to float32.

class singa.autograd.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, group=1, bias=True, pad_mode='NOTSET', **kwargs)

Generate a Conv 2d operator

class singa.autograd.Cos

Calculates the cosine of the given input tensor, element-wise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Cosh

Calculates the hyperbolic cosine of the given input tensor element-wise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.CrossEntropy
backward(dy=1.0)
Parameters

dy (float or CTensor) – scalar, accumulate gradient from outside of current network, usually equal to 1.0

Returns

data for the dL /dx, L is the loss, x is the output

of current network. note that this is true for dy = 1.0

Return type

dx (CTensor)

forward(x, t)
Parameters
  • x (CTensor) – 1d or 2d tensor, the prediction data(output) of current network.

  • t (CTensor) – 1d or 2d tensor, the target data for training.

Returns

scalar.

Return type

loss (CTensor)

class singa.autograd.Div

Performs element-wise binary division (with Numpy-style broadcasting support).

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

a CTensor tuple for (da, db), da is data for dL / da, db is data

for dL / db.

forward(a, b)

Return np.div(a,b), where a and b are CTensor.

class singa.autograd.Dropout(ratio=0.5)

Init a Dropout, which scales the masked input data by the following equation: output = scale * data * mask, scale = 1. / (1. - ratio).

backward(dy)

backward of Dropout :param dy: gradient tensor. :type dy: CTensor

Returns

the gradient tensor over input tensor.

forward(x)

forward of Dropout :param x: input tensor. :type x: CTensor

Returns

the output CTensor.

class singa.autograd.Dummy(tensor, name=None)

Dummy operation whice serves as a placehoder for autograd :param name: set it for debug :type name: string

grad_name(idx)
Parameters

idx – index of the output among all outputs

Returns

the name of the gradient of the output tensor

output_name(idx)
Parameters

idx – index of the output among all outputs

Returns

the name of the output tensor

class singa.autograd.Elu(alpha=1.0)

f(x) = alpha * (exp(x) - 1.) for x < 0, f(x) = x for x >= 0., is applied to the tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – dL / dy

Returns

dL / dx

Return type

dx (CTensor)

forward(x)
Parameters

x (CTensor) – matrix

Returns

a CTensor for the result

class singa.autograd.Equal

Returns the tensor resulted from performing the equal logical operation elementwise on the input tensors x and y.

backward(dy)
Parameters

dy (CTensor) – data for the dL / dy, L is the loss

Raises

AssertionError – no backward function for this operator

forward(x, y)

Return a=b, where a and b are CTensor.

class singa.autograd.Exp(name=None)

y = exp(x), is applied to the tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(a)

Return exp(a), where a is Tensor.

class singa.autograd.Flatten(axis=1)

Flattens the input tensor into a 2D matrix. If input tensor has shape (d_0, d_1, … d_n) then the output will have shape (d_0 X d_1 … d_(axis-1), d_axis X d_(axis+1) … X dn).

backward(dy)
Parameters

dy (CTensor) – data for the dL / dy, L is the loss

Returns

data for the dL / dx, L is the loss,

Return type

dx (CTensor)

forward(x)
Parameters

x (CTensor) – the input tensor

Returns

the result CTensor

class singa.autograd.Gather(axis, indices)

Init a Gather, Given data tensor of rank r >= 1, and indices tensor of rank q, gather entries of the axis dimension of data (by default outer-most one as axis=0) indexed by indices, and concatenates them in an output tensor of rank q + (r - 1).

backward(dy)

backward of Gather :param dy: gradient tensor. :type dy: CTensor

Returns

the gradient tensor over input tensor.

forward(x)

forward of Gather :param x: input tensor. :type x: CTensor

Returns

the output CTensor.

class singa.autograd.Gemm(alpha=1.0, beta=1.0, transA=0, transB=0)

Init a General Matrix multiplication(Gemm) operator. Compute Y = alpha * A’ * B’ + beta * C, where input tensor A has shape (M, K) or (K, M), input tensor B has shape (K, N) or (N, K), input tensor C is broadcastable to shape (M, N), and output tensor Y has shape (M, N). A’ = transpose(A) if transA else A B’ = transpose(B) if transB else B

backward(dy)

backward propogation of Gemm :param dy: The shape of A should be (M, K) if transA is 0, or (K, M) if transA is non-zero. :type dy: CTensor

Returns

CTensor, the gradient over A CTensor, the gradient over B CTensor(optional), the gradient over C

forward(A, B, C=None)

forward propogation of Gemm :param A: The shape of A should be (M, K) if transA is 0, or

(K, M) if transA is non-zero.

Parameters
  • B (CTensor) – The shape of B should be (K, N) if transB is 0, or (N, K) if transB is non-zero.

  • C (CTensor) – (optional), Optional input tensor C. If not specified, the computation is done as if C is a scalar 0. The shape of C should be unidirectional broadcastable to (M, N).

Returns

tensor, the output

class singa.autograd.GlobalAveragePool(data_format='channels_first')

Init a GlobalAveragePool operator

backward(dy)

backward propogation of GlobalAveragePool :param dy: the gradient tensor from upper operations :type dy: CTensor

Returns

CTensor, the gradient over input

forward(x)

forward propogation of GlobalAveragePool :param x: the input tensor :type x: CTensor

Returns

CTensor, the output

class singa.autograd.Greater

Returns the tensor resulted from performing the greater logical operation elementwise on the input tensors A and B.

backward(dy)
Parameters

dy (CTensor) – data for the dL / dy, L is the loss.

Raises

AssertionError – no backward function for this operator.

forward(x, y)

Return a>b, where a and b are CTensor.

class singa.autograd.HardSigmoid(alpha=0.2, gamma=0.5)

y = max(0, min(1, alpha * x + beta)), is applied to the tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – matrix

Returns

a CTensor for the result

class singa.autograd.Identity

Init a identity operator

backward(dy)
Parameters

dy (CTensor) – dL / dy.

Returns

dL / dx.

Return type

dx (CTensor)

forward(x)
Parameters

x (CTensor) – input tensor.

Returns

the same CTensor x.

class singa.autograd.LSTM(input_size, hidden_size, nonlinearity='tanh', num_layers=1, bias=True, batch_first=False, dropout=0, bidirectional=False)

Generate a LSTM operator

class singa.autograd.LeakyRelu(a)

f(x) = alpha * x for x < 0, f(x) = x for x >= 0, is applied to the tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Less

Returns the tensor resulted from performing the less logical operation elementwise on the input CTensors x and y.

backward(dy)
Parameters

dy (CTensor) – data for the dL / dy, L is the loss.

Raises

AssertionError – no backward function for this operator.

forward(x, y)

Return a<b, where a and b are CTensor.

class singa.autograd.Linear(in_features, out_features, bias=True)

Generate a Linear operator

class singa.autograd.Log

y = log(x), is applied to the tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)

Return log(x), where x is CTensor.

class singa.autograd.Matmul

Init matrix multiplication operator.

backward(dy)
Parameters

dy (CTensor) – data for the dL / dy, L is the loss.

Returns

a tuple for (dx, dw).

forward(x, w)

Return np.matmul(x,w), where x and w are CTensor.

class singa.autograd.Max

Element-wise max of each of the input tensors (with Numpy-style broadcasting support).

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

a tuple for (*dx), dx is data for dL / dx.

forward(*x)
Parameters

*x (a list of CTensor) – List of tensors for max.

Returns

CTensor, the output

class singa.autograd.MaxPool1d(kernel_size, stride=None, padding=0, odd_padding=0, 0, 0, 0)

Generate a Max Pooling 1d operator

class singa.autograd.MaxPool2d(kernel_size, stride=None, padding=0, odd_padding=0, 0, 0, 0)

Generate a Max Pooling 2d operator

class singa.autograd.Mean

Element-wise mean of each of the input CTensors.

backward(dy)
Parameters

dy (CTensor) – dL / dy.

Returns

a list of dx (CTensor).

forward(*l)
Parameters

l (a list of CTensor) – a list of CTensor for element-wise mean.

Returns

a new CTensor.

class singa.autograd.MeanSquareError
backward(dy=1.0)

Backward propagation. :param dys: input args consisting of only CTensors.

Returns

CTensor instance(s)

forward(x, t)

Forward propagation. :param xs: input args consisting of only CTensors.

Returns

CTensor instance(s)

class singa.autograd.Min

Element-wise min of each of the input tensors (with Numpy-style broadcasting support).

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

a tuple for (*dx), dx is data for dL / dx.

forward(*x)
Parameters

*x (a list of CTensor) – List of tensors for max.

Returns

CTensor, the output

class singa.autograd.Mul

Performs element-wise binary multiplication (with Numpy-style broadcasting support).

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

a tuple for (da, db), da is data for dL / da, db is data

for dL / db.

forward(a, b)

Return np.multiply(a,b), where a and b are CTensor.

class singa.autograd.Negative

y = -x, is applied to the tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)

Return -x, where x is CTensor.

class singa.autograd.NonZero

Init a NonZero, Constructs a tensor by tiling a given tensor. This is the same as function tile in Numpy: https://docs.scipy.org/doc/numpy/reference/generated/numpy.tile.html

backward(dy)

backward of NonZero :param dy: gradient tensor. :type dy: CTensor

Raises

AssertionError – no backward function for this operator

forward(x)

forward of NonZero :param x: input tensor. :type x: CTensor

Returns

the output CTensor.

class singa.autograd.Not

Returns the negation of the input tensor element-wise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Raises

AssertionError – no backward function for this operator

forward(x)

Return np.logical_not(x), where x is CTensor.

class singa.autograd.OneHot(axis, depth, values)

Produces a one-hot tensor based on inputs.

backward(dy)

backward of OneHot :param dy: gradient tensor. :type dy: CTensor

Raises

AssertionError – no backward function for this operator

forward(indices)

forward of OneHot, we borrow this function from onnx :param indices: Scalar specifying the number of classes in

one-hot tensor. The values in the ‘indices’ input tensor are expected to be in the range [-depth, depth-1].

Returns

the output CTensor.

class singa.autograd.Operation(name=None)

An operation includes the forward and backward function of tensor calculation. Steps to add a specific operation Xxxx: 1. create a subclass of Operation, name it as Xxxx 2. override the forward() and backward(); The arguments of forward()

and backward() should only include CTensor;

backward(*dys)

Backward propagation. :param dys: input args consisting of only CTensors.

Returns

CTensor instance(s)

forward(*xs)

Forward propagation. :param xs: input args consisting of only CTensors.

Returns

CTensor instance(s)

grad_name(idx)
Parameters

idx – index of the output among all outputs

Returns

the name of the gradient of the output tensor

output_name(idx)
Parameters

idx – index of the output among all outputs

Returns

the name of the output tensor

class singa.autograd.Or

Returns the tensor resulted from performing the or logical operation elementwise on the input tensors A and B (with Numpy-style broadcasting support).

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Raises

AssertionError – no backward function for this operator

forward(a, b)

Return np.logical_or(a,b), where a and b are CTensor.

class singa.autograd.PRelu

PRelu applies the function f(x) = slope * x for x < 0, f(x) = x for x >= 0 to the data tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – dL / dy

Returns

dL / dx

Return type

dx (CTensor)

forward(x, slope)
Parameters

x (CTensor) – matrix.

Returns

the result CTensor

class singa.autograd.Pooling2d(kernel_size, stride=None, padding=0, is_max=True, pad_mode='NOTSET')

Generate a Pooling 2d operator

class singa.autograd.Pow

f(x) = a^b, is applied to the tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

a tuple for (da, db), da is data for dL / da, db is data

for dL / db.

forward(a, b)

Return a^b, where a and b are CTensor.

class singa.autograd.RNN(input_size, hidden_size, num_layers=1, nonlinearity='tanh', bias=True, batch_first=False, dropout=0, bidirectional=False)

Generate a RNN operator

class singa.autograd.ReLU

Relu means rectified linear function, i.e, y = max(0, x) is applied to the CTensor elementwise.

backward(dy)
Parameters

dy (CTensor) – dL / dy.

Returns

dL / dx = dy if x >= 0; otherwise 0.

Return type

dx (CTensor)

forward(x)
Parameters

x (CTensor) – input tensor.

Returns

a new CTensor whose element y = x if x >= 0; otherwise 0.

class singa.autograd.Reciprocal

y = 1/x, is applied to the tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)

Return 1/x, where x is CTensor.

class singa.autograd.ReduceMean(axes=None, keepdims=1)

Init a ReduceMean, computes the mean of the input tensor’s element along the provided axes.

backward(dy)

backward of ReduceMean :param dy: gradient tensor. :type dy: CTensor

Returns

the gradient tensor over input tensor.

forward(x)

forward of ReduceMean :param x: input tensor. :type x: CTensor

Returns

the output CTensor.

class singa.autograd.ReduceSum(axes=None, keepdims=1)

Init a ReduceSum, computes the sum of the input tensor’s element along the provided axes.

backward(dy)

backward of ReduceSum :param dy: gradient tensor. :type dy: CTensor

Returns

the gradient tensor over input tensor.

forward(x)

forward of ReduceSum :param x: input tensor. :type x: CTensor

Returns

the output CTensor.

class singa.autograd.Reshape(shape)

Reshape the input tensor similar to np.reshape.

backward(dy)
Parameters

dy (CTensor) – dL / dy

Returns

dL / dx

Return type

dx (CTensor)

forward(x)
Parameters

x (CTensor) – matrix.

Returns

the result CTensor

class singa.autograd.SeLU(alpha=1.67326, gamma=1.0507)

y = gamma * (alpha * e^x - alpha) for x <= 0, y = gamma * x for x > 0 is applied to the tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – dL / dy

Returns

dL / dx

Return type

dx (CTensor)

forward(x)
Parameters

x (CTensor) – matrix

Returns

a CTensor for the result

class singa.autograd.SeparableConv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, bias=False)

Generate a Conv 2d operator

class singa.autograd.Shape

Takes a tensor as input and outputs a tensor containing the shape of the input tensor.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Sigmoid

y = 1 / (1 + exp(-x)), is applied to the tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Sign

Calculate the sign of the given input tensor element-wise. If input > 0, output 1. if input < 0, output -1. if input == 0, output 0.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(a)
Parameters

a (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Sin

Calculates the sine of the given input tensor, element-wise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Sinh

Calculates the hyperbolic sine of the given input tensor element-wise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Slice(starts, ends, axes=None, steps=None)

Init a Slice, Produces a slice of the input tensor along multiple axes. Similar to numpy: https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html

backward(dy)

backward of Slice :param dy: gradient tensor. :type dy: CTensor

Returns

the gradient tensor over input tensor.

forward(x)

forward of Slice :param x: input tensor. :type x: CTensor

Returns

the output CTensor.

class singa.autograd.SoftMax(axis=1)

Apply SoftMax for each row of the Tensor or each column of the Tensor according to the parameter axis.

backward(dy)
Parameters

dy (CTensor) – dL / dy

Returns

dL / dx

Return type

dx (CTensor)

forward(x)
Parameters

x (CTensor) – the input 1d or 2d tensor

Returns

the result CTensor

class singa.autograd.SoftMaxCrossEntropy(t)
backward(dy=1.0)

Backward propagation. :param dys: input args consisting of only CTensors.

Returns

CTensor instance(s)

forward(x)

Forward propagation. :param xs: input args consisting of only CTensors.

Returns

CTensor instance(s)

class singa.autograd.SoftPlus

y = ln(exp(x) + 1) is applied to the tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)

Return ln(exp(x) + 1), where x is CTensor.

class singa.autograd.SoftSign

Calculates the softsign (x/(1+|x|)) of the given input tensor element-wise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)

Return (x/(1+|x|)), where x is CTensor.

class singa.autograd.Split(axis, parts, num_output=None)

Init a Split, Split a tensor into a list of tensors, along the specified ‘axis’.

backward(*dys)

backward of Split :param dys: list of CTensor, gradient tensor.

Returns

the gradient tensor over input tensor.

forward(x)

forward of Split :param x: input tensor. :type x: CTensor

Returns

the output CTensor.

class singa.autograd.Sqrt

y = x^0.5, is applied to the tensor elementwise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)

Return x^0.5, where x is CTensor.

class singa.autograd.Squeeze(axis=[])

Remove single-dimensional entries from the shape of a tensor. Takes a parameter axes with a list of axes to squeeze. If axes is not provided, all the single dimensions will be removed from the shape. If an axis is selected with shape entry not equal to one, an error is raised.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Sub

Performs element-wise binary subtraction (with Numpy-style broadcasting support).

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

a tuple for (da, db), da is data for dL / da, db is data

for dL / db.

forward(a, b)

Return a-b, where x is CTensor.

class singa.autograd.Sum

Element-wise sum of each of the input tensors

backward(dy)
Parameters

dy (CTensor) – dL / dy

Returns

dL / dx

Return type

dx (CTensor)

forward(*l)
Parameters

l (a list of CTensor) – element-wise sum operator

Returns

a CTensor for the result

class singa.autograd.Tan

Insert single-dimensional entries to the shape of an input tensor (data).

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Tanh

Calculates the hyperbolic tangent of the given input tensor element-wise.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Tile(repeats)

Init a Tile, Constructs a tensor by tiling a given tensor. This is the same as function tile in Numpy: https://docs.scipy.org/doc/numpy/reference/generated/numpy.tile.html

backward(dy)

backward of Tile :param dy: gradient tensor. :type dy: CTensor

Returns

the gradient tensor over input tensor.

forward(x)

forward of Tile :param x: input tensor. :type x: CTensor

Returns

the output CTensor.

class singa.autograd.Transpose(perm)

Transpose the input tensor similar to numpy.transpose.

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Unsqueeze(axis)

Insert single-dimensional entries to the shape of an input tensor (data).

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Returns

CTensor, the gradient over input

forward(x)
Parameters

x (CTensor) – Input tensor

Returns

CTensor, the output

class singa.autograd.Xor

Performing the xor logical operation elementwise on the input tensors A and B (with Numpy-style broadcasting support).

backward(dy)
Parameters

dy (CTensor) – the gradient tensor from upper operations

Raises

AssertionError – no backward function for this operator

forward(a, b)

Return np.logical_xor(a,b), where a and b are CTensor.

singa.autograd.abs(a)

Return abs(a), where a is Tensor.

singa.autograd.acos(x)

Calculates the arccosine (inverse of cosine) of the given input tensor, element-wise. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.acosh(x)

Calculates the hyperbolic arccosine of the given input tensor element-wise. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.add(a, b)

Return a+b, where a and b are Tensor.

singa.autograd.add_bias(x, b, axis=0)

Add Bias to each row / column of the Tensor, depending on the axis arg. :param x: matrix. :type x: Tensor :param b: bias to be added. :type b: Tensor :param axis: 0 or 1, default is 0. :type axis: int

Returns

the result Tensor

singa.autograd.asin(x)

Calculates the arcsine (inverse of sine) of the given input tensor, element-wise. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.asinh(x)

Calculates the hyperbolic arcsine of the given input tensor element-wise. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.atan(x)

Calculates the arctangent (inverse of tangent) of the given input tensor, element-wise. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.atanh(x)

Calculates the hyperbolic arctangent of the given input tensor element-wise. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.axis_helper(y_shape, x_shape)

check which axes the x has been broadcasted :param y_shape: the shape of result :param x_shape: the shape of x

Returns

a tuple refering the axes

singa.autograd.back_broadcast(y_shape, x_shape, x)

for a brodcasted tensor, restore its shape of x from y_shape to x_shape :param y_shape: the shape of result :param x_shape: the shape of x :param x: the input

Returns

a tensor

singa.autograd.backward(y, dy=None)

Run the backward propagation starting at y. :param y: a Tensor instance, usually the loss :param dy: a number or a Tensor instance, for the gradient of the

objective/loss w.r.t y, usually None, i.e., 1.0

Returns

yeild the parameter (tensor with stores_grad true) and the

gradient tensors.

singa.autograd.batchnorm_2d(handle, x, scale, bias, running_mean, running_var)

Carries out batch normalization as described in the paper https://arxiv.org/abs/1502.03167. :param handle: BatchNormHandle for cpu and CudnnBatchNormHandle

for gpu

Parameters
  • x (Tensor) – the input tensor

  • scale (Tensor) – the bias tensor

  • bias (Tensor) – the bias tensor

  • running_mean (float) – the running_mean

  • running_var (float) – the running_var

Returns

the result Tensor

singa.autograd.cast(x, to)

The operator casts the elements of a given input tensor to a data type specified by the ‘to’ argument and returns an output tensor of the same size in the converted type. :param x: input tensor. :type x: Tensor :param to: data type, float32 = 0; int = 2. :type to: int

Returns

the output Tensor.

singa.autograd.cat(xs, axis=0)

Concatenate a list of tensors into a single tensor. All input tensors must have the same shape, except for the dimension size of the axis to concatenate on. :param xs: List of tensors for concatenation :type xs: a list of Tensor :param axis: Which axis to concat on. A negative value means

counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(inputs).

Returns

a Tensor for the result

singa.autograd.ceil(x)

Ceil takes one input data (Tensor) and produces one output data (Tensor) where the ceil is, y = ceil(x), is applied to the tensor elementwise. :param x: input tensor. :type x: Tensor

Returns

the output Tensor.

singa.autograd.clip(x, min=None, max=None)

Clip operator limits the given input within an interval. The interval is specified by the inputs ‘min’ and ‘max’. :param x: input tensor :type x: Tensor :param min: Minimum value, under which element is replaced by min. :type min: float :param max: Maximum value, above which element is replaced by max. :type max: float

Returns

a new Tensor with np.clip(x,min,max).

singa.autograd.constant_of_shape(x, value=0)

Init a ConstantOfShape, generate a tensor with given value and shape. :param x: Tensor, 1D tensor. The shape of the expected output tensor.

All values must be >= 0.

Parameters

value (float) – (Optional) The value of the output elements. Should be a one-element value. If not specified, it defaults to 0 and datatype float32

Returns

the output Tensor. If attribute ‘value’ is specified, the value

and datatype of the output tensor is taken from ‘value’. If attribute ‘value’ is not specified, the value in the output defaults to 0, and the datatype defaults to float32.

singa.autograd.conv2d(handle, x, W, b=None, odd_padding=0, 0, 0, 0)

Conv 2d operator :param handle: ConvHandle for cpu or CudnnConvHandle for gpu :type handle: object :param x: input :type x: Tensor :param W: weight :type W: Tensor :param b: bias :type b: Tensor :param odd_padding: , the odd paddding is the value

that cannot be handled by the tuple padding (w, h) mode so we need to firstly handle the input, then use the nomal padding method.

singa.autograd.cos(x)

Calculates the cosine of the given input tensor, element-wise. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.cosh(x)

Calculates the hyperbolic cosine of the given input tensor element-wise. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.ctensor2numpy(x)

To be used in SoftMax Operation. Convert a singa_tensor to numpy_tensor.

singa.autograd.div(a, b)

Return np.div(a,b), where a and b are Tensor.

singa.autograd.dropout(x, ratio=0.5)

Init a Dropout, which scales the masked input data by the following equation: output = scale * data * mask, scale = 1. / (1. - ratio). :param x: input tensor. :type x: Tensor :param ratio: the ratio of random dropout, with value in [0, 1). :type ratio: float

Returns

the output Tensor.

singa.autograd.elu(x, alpha=1)

f(x) = alpha * (exp(x) - 1.) for x < 0, f(x) = x for x >= 0., is applied to the tensor elementwise. :param x: matrix :type x: Tensor :param alpha: Coefficient of ELU, default is 1.0 :type alpha: float

Returns

a Tensor for the result

singa.autograd.equal(x, y)

Return a=b, where a and b are Tensor.

singa.autograd.exp(a)

Return exp(a), where a is Tensor.

singa.autograd.flatten(x, axis=1)

Flattens the input tensor into a 2D matrix. If input tensor has shape (d_0, d_1, … d_n) then the output will have shape (d_0 X d_1 … d_(axis-1), d_axis X d_(axis+1) … X dn). :param x: the input tensor :type x: Tensor :param axis: Indicate up to which input dimensions (exclusive)

should be flattened to the outer dimension of the output. The value for axis must be in the range [-r, r], where r is the rank of the input tensor. Negative value means counting dimensions from the back. When axis = 0, the shape of the output tensor is (1, (d_0 X d_1 … d_n), where the shape of the input tensor is (d_0, d_1, … d_n).

Returns

the result Tensor

singa.autograd.gather(x, axis, indices)

Init a Gather, Given data tensor of rank r >= 1, and indices tensor of rank q, gather entries of the axis dimension of data (by default outer-most one as axis=0) indexed by indices, and concatenates them in an output tensor of rank q + (r - 1). :param x: input tensor. :type x: Tensor :param axis: which axis to slice on. A negative value means counting

dimensions from the back. Accepted range is [-rank, rank-1] where r = rank(input).

Parameters

indices (list of int) – entries of the axis dimension of data.

Returns

the output Tensor.

singa.autograd.gemm(A, B, C=None, alpha=1.0, beta=1.0, transA=0, transB=0)

Init a General Matrix multiplication(Gemm) operator. Compute Y = alpha * A’ * B’ + beta * C, where input tensor A has shape (M, K) or (K, M), input tensor B has shape (K, N) or (N, K), input tensor C is broadcastable to shape (M, N), and output tensor Y has shape (M, N). A’ = transpose(A) if transA else A B’ = transpose(B) if transB else B :param A: The shape of A should be (M, K) if transA is 0, or

(K, M) if transA is non-zero.

Parameters
  • B (Tensor) – The shape of B should be (K, N) if transB is 0, or (N, K) if transB is non-zero.

  • C (Tensor) – (optional), Optional input tensor C. If not specified, the computation is done as if C is a scalar 0. The shape of C should be unidirectional broadcastable to (M, N).

  • alpha (float) – Scalar multiplier for the product of input tensors A * B.

  • beta (float) – Scalar multiplier for input tensor C.

  • ransA (int) – Whether A should be transposed

  • transB (int) – Whether B should be transposed

Returns

Tensor, the output

singa.autograd.globalaveragepool(x, data_format='channels_first')

GlobalAveragePool operator :param x: the input tensor :type x: Tensor :param data_format: A string, we support two formats:

channels_last and channels_first, default is channels_first. channels_first means the format of input is (N x C x H x W) channels_last means the format of input is (N x H x W x C)

Returns

Tensor, the output

singa.autograd.gradients(y, dy=None)

Compute the gradients of the output w.r.t the parameters

Parameters
  • y – the output tensor, e.g., the loss

  • dy – gradient of the target w.r.t y; None indicates the gradient is 1.0; it can be used to rescale the loss.

Returns

a dictionary storing the gradient tensors of all tensors

whose stores_grad is true (e.g. parameter tensors)

singa.autograd.greater(x, y)

Return a>b, where a and b are Tensor.

singa.autograd.hardsigmoid(x, alpha=0.2, gamma=0.5)

y = max(0, min(1, alpha * x + beta)), is applied to the tensor elementwise. :param x: matrix :type x: Tensor :param alpha: Value of alpha. :type alpha: float :param gamma: Value of beta. :type gamma: float

Returns

a Tensor for the result

singa.autograd.identity(x)

Init a identity operator. :param x: input tensor. :type x: Tensor

Returns

the same Tensor with x.

singa.autograd.infer_dependency(op)

Infer the dependency of all operations with the given op as the last operation. Operation A is depending on B if A uses the output(s) of B.

Parameters

op – an Operation instance, e.g. the loss operation.

Returns

a Counter instance with the operation as the key, and the number of operations that are depending on it as the value; and a Counter instance with the id of the output tensor as the key, and the number of operations that are depending on it as the value.

singa.autograd.leakyrelu(x, a=0.01)

f(x) = alpha * x for x < 0, f(x) = x for x >= 0 is applied to the tensor elementwise. :param x: Input tensor :type x: Tensor :param a: Coefficient of leakage, default to 0.01. :type a: float

Returns

Tensor, the output

singa.autograd.less(x, y)

Return a<b, where a and b are CTensor.

singa.autograd.log(x)

Return log(x), where x is Tensor.

singa.autograd.matmul(x, w)

Return np.matmul(x,w), where x and w are Tensor.

singa.autograd.max(*l)

Element-wise max of each of the input tensors (with Numpy-style broadcasting support). :param *x: List of tensors for max. :type *x: a list of Tensor

Returns

CTensor, the output

singa.autograd.mean(*l)

Element-wise mean of each of the input tensors. :param l: element-wise mean operator. :type l: a list of Tensor

Returns

a new Tensor.

singa.autograd.min(*l)

Element-wise min of each of the input tensors (with Numpy-style broadcasting support). :param *x: List of tensors for max. :type *x: a list of Tensor

Returns

Tensor, the output

singa.autograd.mul(x, y)

Return np.multiply(x,y), where a and b are Tensor.

singa.autograd.negative(x)

Return -x, where x is Tensor.

singa.autograd.nonzero(x)

Init a NonZero, Constructs a tensor by tiling a given tensor. This is the same as function tile in Numpy: https://docs.scipy.org/doc/numpy/reference/generated/numpy.tile.html :param x: input tensor. :type x: Tensor

Returns

the output Tensor.

singa.autograd.onehot(axis, indices, depth, values)

Produces a one-hot tensor based on inputs. :param axis: Axis along which one-hot representation in added.

Default: axis=-1. axis=-1 means that the additional dimension will be inserted as the innermost/last dimension in the output tensor.

Parameters
  • indices (Tensor) – Scalar specifying the number of classes in one-hot tensor. The values in the ‘indices’ input tensor are expected to be in the range [-depth, depth-1].

  • depth (int) – Scalar specifying the number of classes in one-hot tensor. This is also the size of the one-hot dimension (specified by ‘axis’ attribute) added on in the output tensor. The values in the ‘indices’ input tensor are expected to be in the range [-depth, depth-1].

  • values (float) – Rank 1 tensor containing exactly two elements, in the format [off_value, on_value], where ‘on_value’ is the value used for filling locations specified in ‘indices’ input tensor,

Returns

the output Tensor.

singa.autograd.pooling_2d(handle, x, odd_padding=0, 0, 0, 0)

Pooling 2d operator :param handle: PoolingHandle for cpu or CudnnPoolingHandle for

gpu

Parameters
  • x (Tensor) – input

  • odd_padding (tuple of four int) – the odd paddding is the value that cannot be handled by the tuple padding (w, h) mode so it needs to firstly handle the input, then use the normal padding method.

Returns

the result Tensor

singa.autograd.pow(a, b)

Return a^b, where a and b are Tensor.

singa.autograd.prelu(x, slope)

PRelu applies the function f(x) = slope * x for x < 0, f(x) = x for x >= 0 to the data tensor elementwise. :param x: matrix. :type x: Tensor

Returns

the result Tensor

singa.autograd.reciprocal(x)

Return 1/x, where x is Tensor.

singa.autograd.reduce_mean(x, axes=None, keepdims=1)

Init a ReduceMean, computes the mean of the input tensor’s element along the provided axes. :param x: input tensor. :type x: Tensor :param axes: A list of integers, along which to reduce.

Accepted range is [-r, r-1] where r = rank(data). The default is None, which reduces over all the dimensions of the input tensor.

Parameters

keepdims (int) – Keep the reduced dimension or not, default 1 mean keep reduced dimension.

Returns

the output Tensor.

singa.autograd.reduce_sum(x, axes=None, keepdims=1)

Init a ReduceSum, computes the sum of the input tensor’s element along the provided axes. :param x: input tensor. :type x: Tensor :param axes: A list of integers, along which to reduce.

Accepted range is [-r, r-1] where r = rank(data). The default is None, which reduces over all the dimensions of the input tensor.

Parameters

keepdims (int) – Keep the reduced dimension or not, default 1 mean keep reduced dimension.

Returns

the output Tensor.

singa.autograd.relu(x)

Relu means rectified linear function, i.e, y = max(0, x) is applied to the CTensors elementwise. :param x: input tensor. :type x: Tensor

Returns

a new Tensor whose element y = x if x >= 0; otherwise 0.

singa.autograd.reshape(x, shape)

Reshape the input tensor similar to mp.reshape. :param x: matrix. :type x: Tensor :param shape: Specified shape for output. At most one

dimension of the new shape can be -1. In this case, the value is inferred from the size of the tensor and the remaining dimensions. A dimension could also be 0, in which case the actual dimension value is unchanged (i.e. taken from the input tensor).

Returns

the result Tensor

singa.autograd.selu(x, alpha=1.67326, gamma=1.0507)

y = gamma * (alpha * e^x - alpha) for x <= 0, y = gamma * x for x > 0 is applied to the tensor elementwise. :param x: matrix :type x: Tensor :param alpha: Coefficient of SELU default to 1.67326 :type alpha: float :param gamma: Coefficient of SELU default to 1.0507 :type gamma: float

Returns

a Tensor for the result

singa.autograd.shape(x)

Takes a tensor as input and outputs a tensor containing the shape of the input tensor. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.sigmoid(x)

y = 1 / (1 + exp(-x)), is applied to the tensor elementwise. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.sign(a)

Calculate the sign of the given input tensor element-wise. If input > 0, output 1. if input < 0, output -1. if input == 0, output 0. :param a: Input tensor :type a: Tensor

Returns

Tensor, the output

singa.autograd.sin(x)

Calculates the sine of the given input tensor, element-wise. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.sinh(x)

Calculates the hyperbolic sine of the given input tensor element-wise. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.slice(x, starts, ends, axes=None, steps=None)

Init a Slice, Produces a slice of the input tensor along multiple axes. Similar to numpy: https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html :param x: input tensor. :type x: Tensor :param starts: starting indices of corresponding axis :type starts: list of int :param ends: ending indices of corresponding axis :type ends: list of int :param axes: axes that starts and ends apply to.

Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(data).

Parameters

steps (list of int) – slice step of corresponding axis in axes. Negative value means slicing backward. ‘steps’ cannot be 0. Defaults to 1.

Returns

the output Tensor.

singa.autograd.softmax(x, axis=1)

Apply SoftMax for each row of the Tensor or each column of the Tensor according to the parameter axis. :param x: the input 1d or 2d tensor :type x: Tensor :param axis: axis of softmax, default to 1 :type axis: int

Returns

the result Tensor

singa.autograd.softplus(x)

Return ln(exp(x) + 1), where x is Tensor.

singa.autograd.softsign(x)

Return (x/(1+|x|)), where x is Tensor.

singa.autograd.split(x, axis, parts, num_output=None)

Init a Split, Split a tensor into a list of tensors, along the specified ‘axis’. :param x: input tensor. :type x: Tensor :param axis: which axis to split on. A negative value means

counting dimensions from the back. Accepted range is [-rank, rank-1] where r = rank(input).

Parameters
  • parts (list of int) – length of each output, which can be specified using argument ‘parts’. Otherwise, the tensor is parts to equal sized parts.

  • num_output (bool) – once parts is none, the tensor is split to equal sized parts for each output.

Returns

the output Tensor.

singa.autograd.sqrt(x)

Return x^0.5, where x is Tensor.

singa.autograd.squeeze(x, axis=[])

Remove single-dimensional entries from the shape of a tensor. Takes a parameter axes with a list of axes to squeeze. If axes is not provided, all the single dimensions will be removed from the shape. If an axis is selected with shape entry not equal to one, an error is raised. :param x: Input tensor :type x: Tensor :param axis: List of integers indicating the dimensions

to squeeze. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(data).

Returns

Tensor, the output

singa.autograd.sub(a, b)

Return a-b, where a and b are Tensor.

singa.autograd.sum(*l)

Element-wise sum of each of the input tensors :param l: element-wise sum operator :type l: a list of Tensor

Returns

a Tensor for the result

singa.autograd.tan(x)

Calculates the tangent of the given input tensor, element-wise. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.tanh(x)

Calculates the hyperbolic tangent of the given input tensor element-wise. :param x: Input tensor :type x: Tensor

Returns

Tensor, the output

singa.autograd.tile(x, repeats)

Init a Tile, Constructs a tensor by tiling a given tensor. This is the same as function tile in Numpy: https://docs.scipy.org/doc/numpy/reference/generated/numpy.tile.html :param x: input tensor. :type x: Tensor :param repeats: 1D int matrix of the same length as input’s

dimension number, includes numbers of repeated copies along input’s dimensions.

Returns

the output Tensor.

singa.autograd.transpose(x, shape)

Transpose the input tensor similar to numpy.transpose. :param x: Input tensor :type x: Tensor :param perm: A list of integers. By default, reverse the

dimensions, otherwise permute the axes according to the values given.

Returns

Tensor, the output

singa.autograd.unsqueeze(x, axis=- 1)

Insert single-dimensional entries to the shape of an input tensor (data). :param x: Input tensor :type x: Tensor :param axis: the dimensions to be inserted. :type axis: list of int

Returns

Tensor, the output