nfnets package

Submodules

nfnets.base module

class nfnets.base.ScaledStdConv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, gain=True, gamma=1.0, eps=1e-05, use_layernorm=False)

Bases: Conv2d

Conv2d layer with Scaled Weight Standardization. Paper: Characterizing signal propagation to close the performance gap in unnormalized ResNets -

https://arxiv.org/abs/2101.08692

Adapted from timm: https://github.com/rwightman/pytorch-image-models/blob/4ea593196414684d2074cbb81d762f3847738484/timm/models/layers/std_conv.py

bias: Optional[Tensor]

dilation: Tuple[int, ...]

forward(x)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

get_weight()

groups: int

kernel_size: Tuple[int, ...]

out_channels: int

output_padding: Tuple[int, ...]

padding: Union[str, Tuple[int, ...]]

padding_mode: str

stride: Tuple[int, ...]

transposed: bool

weight: Tensor

class nfnets.base.WSConv1d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros')

Bases: Conv1d

Applies a 1D convolution over an input signal composed of several input planes. In the simplest case, the output value of the layer with input size \((N, C_{\text{in}}, L)\) and output \((N, C_{\text{out}}, L_{\text{out}})\) can be precisely described as: .. math:

\text{out}(N_i, C_{\text{out}_j}) = \text{bias}(C_{\text{out}_j}) +
\sum_{k = 0}^{C_{in} - 1} \text{weight}(C_{\text{out}_j}, k)
\star \text{input}(N_i, k)

where \(\star\) is the valid cross-correlation operator, \(N\) is a batch size, \(C\) denotes a number of channels, \(L\) is a length of signal sequence. This module supports TensorFloat32. * stride controls the stride for the cross-correlation, a single

number or a one-element tuple.

• padding controls the amount of implicit zero-paddings on both sides for padding number of points.

• dilation controls the spacing between the kernel points; also known as the à trous algorithm. It is harder to describe, but this link has a nice visualization of what dilation does.

• groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,

  • At groups=1, all inputs are convolved to all outputs.

  • At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated.

  • At groups= in_channels, each input channel is convolved with its own set of filters, of size \(\left\lfloor\frac{out\_channels}{in\_channels}\right\rfloor\).

Note

Depending of the size of your kernel, several (of the last) columns of the input might be lost, because it is a valid cross-correlation, and not a full cross-correlation. It is up to the user to add proper padding.

Note

When groups == in_channels and out_channels == K * in_channels, where K is a positive integer, this operation is also termed in literature as depthwise convolution. In other words, for an input of size \((N, C_{in}, L_{in})\), a depthwise convolution with a depthwise multiplier K, can be constructed by arguments \((C_\text{in}=C_{in}, C_\text{out}=C_{in} \times K, ..., \text{groups}=C_{in})\).

Note

In some circumstances when using the CUDA backend with CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. Please see the notes on /notes/randomness for background.

Parameters

• in_channels (int) – Number of channels in the input image

• out_channels (int) – Number of channels produced by the convolution

• kernel_size (int or tuple) – Size of the convolving kernel

• stride (int or tuple, optional) – Stride of the convolution. Default: 1

• padding (int or tuple, optional) – Zero-padding added to both sides of the input. Default: 0

• padding_mode (string, optional) – 'zeros', 'reflect', 'replicate' or 'circular'. Default: 'zeros'

• dilation (int or tuple, optional) – Spacing between kernel elements. Default: 1

• groups (int, optional) – Number of blocked connections from input channels to output channels. Default: 1

• bias (bool, optional) – If True, adds a learnable bias to the output. Default: True

Shape:

• Input: \((N, C_{in}, L_{in})\)

• Output: \((N, C_{out}, L_{out})\) where .. math:

  L_{out} = \left\lfloor\frac{L_{in} + 2 \times \text{padding} - \text{dilation}
            \times (\text{kernel\_size} - 1) - 1}{\text{stride}} + 1\right\rfloor
  

weight

the learnable weights of the module of shape \((\text{out\_channels}, \frac{\text{in\_channels}}{\text{groups}}, \text{kernel\_size})\). The values of these weights are sampled from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) where \(k = \frac{groups}{C_\text{in} * \text{kernel\_size}}\)

Type

Tensor

bias

the learnable bias of the module of shape (out_channels). If bias is True, then the values of these weights are sampled from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) where \(k = \frac{groups}{C_\text{in} * \text{kernel\_size}}\)

Type

Tensor

Examples::

>>> m = nn.Conv1d(16, 33, 3, stride=2)
>>> input = torch.randn(20, 16, 50)
>>> output = m(input)

bias: Optional[Tensor]

dilation: Tuple[int, ...]

forward(input, eps=0.0001)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

groups: int

kernel_size: Tuple[int, ...]

out_channels: int

output_padding: Tuple[int, ...]

padding: Union[str, Tuple[int, ...]]

padding_mode: str

standardize_weight(eps)

stride: Tuple[int, ...]

transposed: bool

weight: Tensor

class nfnets.base.WSConv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros')

Bases: Conv2d

Applies a 2D convolution over an input signal composed of several input

planes after weight normalization/standardization.

Reference: https://github.com/deepmind/deepmind-research/blob/master/nfnets/base.py#L121

In the simplest case, the output value of the layer with input size \((N, C_{ ext{in}}, H, W)\) and output \((N, C_{ ext{out}}, H_{ ext{out}}, W_{ ext{out}})\) can be precisely described as:

\[ext{out}(N_i, C_{ ext{out}_j}) = ext{bias}(C_{ ext{out}_j}) + \sum_{k = 0}^{C_{ ext{in}} - 1} ext{weight}(C_{ ext{out}_j}, k) \star ext{input}(N_i, k)\]

where \(\star\) is the valid 2D cross-correlation operator, \(N\) is a batch size, \(C\) denotes a number of channels, \(H\) is a height of input planes in pixels, and \(W\) is width in pixels.

This module supports TensorFloat32.

• stride controls the stride for the cross-correlation, a single number or a tuple.

• padding controls the amount of implicit zero-paddings on both sides for padding number of points for each dimension.

• dilation controls the spacing between the kernel points; also known as the à trous algorithm. It is harder to describe, but this link has a nice visualization of what dilation does.

• groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,

  • At groups=1, all inputs are convolved to all outputs.

  • At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated.

  • At groups= in_channels, each input channel is convolved with its own set of filters, of size: :math:`leftlfloor

rac{out_channels}{in_channels} ight floor`.

The parameters kernel_size, stride, padding, dilation can either be:

• a single int – in which case the same value is used for the height and width dimension

• a tuple of two ints – in which case, the first int is used for the height dimension, and the second int for the width dimension

Note:

Depending of the size of your kernel, several (of the last) columns of the input might be lost, because it is a valid cross-correlation, and not a full cross-correlation. It is up to the user to add proper padding.

Note:

When groups == in_channels and out_channels == K * in_channels, where K is a positive integer, this operation is also termed in literature as depthwise convolution.

In other words, for an input of size \((N, C_{in}, H_{in}, W_{in})\), a depthwise convolution with a depthwise multiplier K, can be constructed by arguments \((in\_channels=C_{in}, out\_channels=C_{in} imes K, ..., groups=C_{in})\).

Note:

In some circumstances when using the CUDA backend with CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. Please see the notes on /notes/randomness for background.

Args:

in_channels (int): Number of channels in the input image out_channels (int): Number of channels produced by the convolution kernel_size (int or tuple): Size of the convolving kernel stride (int or tuple, optional): Stride of the convolution. Default: 1 padding (int or tuple, optional): Zero-padding added to both sides of

the input. Default: 0

padding_mode (string, optional): 'zeros', 'reflect',

'replicate' or 'circular'. Default: 'zeros'

dilation (int or tuple, optional): Spacing between kernel elements. Default: 1 groups (int, optional): Number of blocked connections from input

channels to output channels. Default: 1

bias (bool, optional): If True, adds a learnable bias to the

output. Default: True

Shape:

• Input: \((N, C_{in}, H_{in}, W_{in})\)

• Output: \((N, C_{out}, H_{out}, W_{out})\) where

  \[H_{out} = \left\lfloor\]

rac{H_{in} + 2 imes ext{padding}[0] - ext{dilation}[0]

imes ( ext{kernel_size}[0] - 1) - 1}{ ext{stride}[0]} + 1

ight floor

\[W_{out} = \left\lfloor\]

rac{W_{in} + 2 imes ext{padding}[1] - ext{dilation}[1]

imes ( ext{kernel_size}[1] - 1) - 1}{ ext{stride}[1]} + 1

ight floor

Attributes:

weight (Tensor): the learnable weights of the module of shape

:math:`( ext{out_channels},

rac{ ext{in_channels}}{ ext{groups}},`

:math:` ext{kernel_size[0]}, ext{kernel_size[1]})`. The values of these weights are sampled from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) where :math:`k =

rac{groups}{C_ ext{in} * prod_{i=0}^{1} ext{kernel_size}[i]}`

bias (Tensor): the learnable bias of the module of shape

(out_channels). If bias is True, then the values of these weights are sampled from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) where :math:`k =

rac{groups}{C_ ext{in} * prod_{i=0}^{1} ext{kernel_size}[i]}`

Examples:

>>> # With square kernels and equal stride
>>> m = WSConv2d(16, 33, 3, stride=2)
>>> # non-square kernels and unequal stride and with padding
>>> m = WSConv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2))
>>> # non-square kernels and unequal stride and with padding and dilation
>>> m = WSConv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2), dilation=(3, 1))
>>> input = torch.randn(20, 16, 50, 100)
>>> output = m(input)

bias: Optional[Tensor]

dilation: Tuple[int, ...]

forward(input, eps=0.0001)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

groups: int

kernel_size: Tuple[int, ...]

out_channels: int

output_padding: Tuple[int, ...]

padding: Union[str, Tuple[int, ...]]

padding_mode: str

standardize_weight(eps)

stride: Tuple[int, ...]

transposed: bool

weight: Tensor

class nfnets.base.WSConvTranspose2d(in_channels: int, out_channels: int, kernel_size, stride=1, padding=0, output_padding=0, groups: int = 1, bias: bool = True, dilation: int = 1, padding_mode: str = 'zeros')

Bases: ConvTranspose2d

Applies a 2D transposed convolution operator over an input image

composed of several input planes after weight normalization/standardization.

This module can be seen as the gradient of Conv2d with respect to its input. It is also known as a fractionally-strided convolution or a deconvolution (although it is not an actual deconvolution operation).

This module supports TensorFloat32.

• stride controls the stride for the cross-correlation.

• padding controls the amount of implicit zero-paddings on both sides for dilation * (kernel_size - 1) - padding number of points. See note below for details.

• output_padding controls the additional size added to one side of the output shape. See note below for details.

• dilation controls the spacing between the kernel points; also known as the à trous algorithm. It is harder to describe, but this link has a nice visualization of what dilation does.

• groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,

  • At groups=1, all inputs are convolved to all outputs.

  • At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated.

  • At groups= in_channels, each input channel is convolved with its own set of filters (of size :math:`leftlfloor

rac{out_channels}{in_channels} ight floor`).

The parameters kernel_size, stride, padding, output_padding can either be:

• a single int – in which case the same value is used for the height and width dimensions

• a tuple of two ints – in which case, the first int is used for the height dimension, and the second int for the width dimension

Note

Depending of the size of your kernel, several (of the last) columns of the input might be lost, because it is a valid cross-correlation, and not a full cross-correlation. It is up to the user to add proper padding.

Note:

The padding argument effectively adds dilation * (kernel_size - 1) - padding amount of zero padding to both sizes of the input. This is set so that when a Conv2d and a ConvTranspose2d are initialized with same parameters, they are inverses of each other in regard to the input and output shapes. However, when stride > 1, Conv2d maps multiple input shapes to the same output shape. output_padding is provided to resolve this ambiguity by effectively increasing the calculated output shape on one side. Note that output_padding is only used to find output shape, but does not actually add zero-padding to output.

Note:

In some circumstances when using the CUDA backend with CuDNN, this operator may select a nondeterministic algorithm to increase performance. If this is undesirable, you can try to make the operation deterministic (potentially at a performance cost) by setting torch.backends.cudnn.deterministic = True. Please see the notes on /notes/randomness for background.

Args:

in_channels (int): Number of channels in the input image out_channels (int): Number of channels produced by the convolution kernel_size (int or tuple): Size of the convolving kernel stride (int or tuple, optional): Stride of the convolution. Default: 1 padding (int or tuple, optional): dilation * (kernel_size - 1) - padding zero-padding

will be added to both sides of each dimension in the input. Default: 0

output_padding (int or tuple, optional): Additional size added to one side

of each dimension in the output shape. Default: 0

groups (int, optional): Number of blocked connections from input channels to output channels. Default: 1 bias (bool, optional): If True, adds a learnable bias to the output. Default: True dilation (int or tuple, optional): Spacing between kernel elements. Default: 1

Shape:

• Input: \((N, C_{in}, H_{in}, W_{in})\)

• Output: \((N, C_{out}, H_{out}, W_{out})\) where

\[H_{out} = (H_{in} - 1) imes ext{stride}[0] - 2 imes ext{padding}[0] + ext{dilation}[0] imes ( ext{kernel\_size}[0] - 1) + ext{output\_padding}[0] + 1\]

\[W_{out} = (W_{in} - 1) imes ext{stride}[1] - 2 imes ext{padding}[1] + ext{dilation}[1] imes ( ext{kernel\_size}[1] - 1) + ext{output\_padding}[1] + 1\]

Attributes:

weight (Tensor): the learnable weights of the module of shape

:math:`( ext{in_channels},

rac{ ext{out_channels}}{ ext{groups}},`

:math:` ext{kernel_size[0]}, ext{kernel_size[1]})`. The values of these weights are sampled from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) where :math:`k =

rac{groups}{C_ ext{out} * prod_{i=0}^{1} ext{kernel_size}[i]}`

bias (Tensor): the learnable bias of the module of shape (out_channels)

If bias is True, then the values of these weights are sampled from \(\mathcal{U}(-\sqrt{k}, \sqrt{k})\) where :math:`k =

rac{groups}{C_ ext{out} * prod_{i=0}^{1} ext{kernel_size}[i]}`

Examples:

>>> # With square kernels and equal stride
>>> m = WSConvTranspose2d(16, 33, 3, stride=2)
>>> # non-square kernels and unequal stride and with padding
>>> m = WSConvTranspose2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2))
>>> input = torch.randn(20, 16, 50, 100)
>>> output = m(input)
>>> # exact output size can be also specified as an argument
>>> input = torch.randn(1, 16, 12, 12)
>>> downsample = WSConv2d(16, 16, 3, stride=2, padding=1)
>>> upsample = WSConvTranspose2d(16, 16, 3, stride=2, padding=1)
>>> h = downsample(input)
>>> h.size()
torch.Size([1, 16, 6, 6])
>>> output = upsample(h, output_size=input.size())
>>> output.size()
torch.Size([1, 16, 12, 12])

bias: Optional[Tensor]

dilation: Tuple[int, ...]

forward(input: Tensor, output_size: Optional[List[int]] = None, eps: float = 0.0001) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

groups: int

kernel_size: Tuple[int, ...]

out_channels: int

output_padding: Tuple[int, ...]

padding: Union[str, Tuple[int, ...]]

padding_mode: str

standardize_weight(eps)

stride: Tuple[int, ...]

transposed: bool

weight: Tensor

nfnets.sgd_agc module

class nfnets.sgd_agc.SGD_AGC(params, lr=<required parameter>, momentum=0, dampening=0, weight_decay=0, nesterov=False, clipping=0.01, eps=0.001)

Bases: Optimizer

Implements stochastic gradient descent (optionally with momentum).

Nesterov momentum is based on the formula from `On the importance of initialization and momentum in deep learning`__ AGC from NFNets: https://arxiv.org/abs/2102.06171.pdf.

Parameters

• params (iterable) – iterable of parameters to optimize or dicts defining parameter groups

• lr (float) – learning rate

• momentum (float, optional) – momentum factor (default: 0)

• weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)

• dampening (float, optional) – dampening for momentum (default: 0)

• nesterov (bool, optional) – enables Nesterov momentum (default: False)

• dampening – dampening for momentum (default: 0.01)

• clipping (float, optional) – clipping value (default: 1e-3)

• eps (float, optional) – eps (default: 1e-3)

Example

>>> optimizer = torch.optim.SGD_AGC(model.parameters(), lr=0.1, momentum=0.9)
>>> optimizer.zero_grad()
>>> loss_fn(model(input), target).backward()
>>> optimizer.step()

Note

The implementation has been adapted from the PyTorch framework and the official NF-Nets paper. The implementation of SGD with Momentum/Nesterov subtly differs from Sutskever et. al. and implementations in some other frameworks.

Considering the specific case of Momentum, the update can be written as

\[\begin{split}\begin{aligned} v_{t+1} & = \mu * v_{t} + g_{t+1}, \\ p_{t+1} & = p_{t} - \text{lr} * v_{t+1}, \end{aligned}\end{split}\]

where \(p\), \(g\), \(v\) and \(\mu\) denote the parameters, gradient, velocity, and momentum respectively.

This is in contrast to Sutskever et. al. and other frameworks which employ an update of the form

\[\begin{split}\begin{aligned} v_{t+1} & = \mu * v_{t} + \text{lr} * g_{t+1}, \\ p_{t+1} & = p_{t} - v_{t+1}. \end{aligned}\end{split}\]

The Nesterov version is analogously modified.

step(closure=None)

Performs a single optimization step.

Parameters

closure (callable, optional) – A closure that reevaluates the model and returns the loss.

nfnets.agc module

class nfnets.agc.AGC(params, optim: Optimizer, clipping: float = 0.01, eps: float = 0.001, model=None, ignore_agc=['fc'])

Bases: Optimizer

Generic implementation of the Adaptive Gradient Clipping

Parameters

• params (iterable) – iterable of parameters to optimize or dicts defining parameter groups

• optim (torch.optim.Optimizer) – Optimizer with base class optim.Optimizer

• clipping (float, optional) – clipping value (default: 1e-3)

• eps (float, optional) – eps (default: 1e-3)

• model (torch.nn.Module, optional) – The original model

• ignore_agc (str, Iterable, optional) – Layers for AGC to ignore

step(closure=None)

Performs a single optimization step.

Parameters

closure (callable, optional) – A closure that reevaluates the model and returns the loss.

zero_grad(set_to_none: bool = False)

Sets the gradients of all optimized torch.Tensor s to zero.

Parameters

set_to_none (bool) – instead of setting to zero, set the grads to None. This is will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests zero_grad(set_to_none=True) followed by a backward pass, .grads are guaranteed to be None for params that did not receive a gradient. 3. torch.optim optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether).

nfnets.utils module

nfnets.utils.replace_conv(module: ~torch.nn.modules.module.Module, conv_class=<class 'nfnets.base.WSConv2d'>)

Recursively replaces every convolution with WSConv2d.

Usage: replace_conv(model) #(In-line replacement) :param module: target’s model whose convolutions must be replaced. :type module: nn.Module :param conv_class: Class of Conv(WSConv2d or ScaledStdConv2d) :type conv_class: Class

nfnets.utils.unitwise_norm(x: Tensor)

nfnets.models.resnet module

nfnets.models.resnet.nf_resnet101(alpha: float = 0.2, beta: float = 1.0, activation: str = 'relu', base_conv: ~torch.nn.modules.conv.Conv2d = <class 'nfnets.base.ScaledStdConv2d'>, **kwargs: ~typing.Any) → NFResNet

ResNet-101 model from “Deep Residual Learning for Image Recognition”. and “High-Performance Large-Scale Image Recognition Without Normalization” <https://arxiv.org/pdf/2102.06171v1>. :param pretrained: If True, returns a model pre-trained on ImageNet :type pretrained: bool

nfnets.models.resnet.nf_resnet152(alpha: float = 0.2, beta: float = 1.0, activation: str = 'relu', base_conv: ~torch.nn.modules.conv.Conv2d = <class 'nfnets.base.ScaledStdConv2d'>, **kwargs: ~typing.Any) → NFResNet

ResNet-152 model from “Deep Residual Learning for Image Recognition”. and “High-Performance Large-Scale Image Recognition Without Normalization” <https://arxiv.org/pdf/2102.06171v1>. :param pretrained: If True, returns a model pre-trained on ImageNet :type pretrained: bool

nfnets.models.resnet.nf_resnet18(alpha: float = 0.2, beta: float = 1.0, activation: str = 'relu', base_conv: ~torch.nn.modules.conv.Conv2d = <class 'nfnets.base.ScaledStdConv2d'>, **kwargs: ~typing.Any) → NFResNet

ResNet-18 model from “Deep Residual Learning for Image Recognition” and “High-Performance Large-Scale Image Recognition Without Normalization” <https://arxiv.org/pdf/2102.06171v1>. :param pretrained: If True, returns a model pre-trained on ImageNet :type pretrained: bool

nfnets.models.resnet.nf_resnet34(alpha: float = 0.2, beta: float = 1.0, activation: str = 'relu', base_conv: ~torch.nn.modules.conv.Conv2d = <class 'nfnets.base.ScaledStdConv2d'>, **kwargs: ~typing.Any) → NFResNet

ResNet-34 model from “Deep Residual Learning for Image Recognition” and “High-Performance Large-Scale Image Recognition Without Normalization” <https://arxiv.org/pdf/2102.06171v1>. :param pretrained: If True, returns a model pre-trained on ImageNet :type pretrained: bool

nfnets.models.resnet.nf_resnet50(alpha: float = 0.2, beta: float = 1.0, activation: str = 'relu', base_conv: ~torch.nn.modules.conv.Conv2d = <class 'nfnets.base.ScaledStdConv2d'>, **kwargs: ~typing.Any) → NFResNet

ResNet-50 model from “Deep Residual Learning for Image Recognition” and “High-Performance Large-Scale Image Recognition Without Normalization” <https://arxiv.org/pdf/2102.06171v1>. :param pretrained: If True, returns a model pre-trained on ImageNet :type pretrained: bool

nfnets.models.resnet.nf_resnext101_32x8d(alpha: float = 0.2, beta: float = 1.0, activation: str = 'relu', base_conv: ~torch.nn.modules.conv.Conv2d = <class 'nfnets.base.ScaledStdConv2d'>, **kwargs: ~typing.Any) → NFResNet

ResNeXt-101 32x8d model from “Aggregated Residual Transformation for Deep Neural Networks”. and “High-Performance Large-Scale Image Recognition Without Normalization” <https://arxiv.org/pdf/2102.06171v1>. :param pretrained: If True, returns a model pre-trained on ImageNet :type pretrained: bool

nfnets.models.resnet.nf_resnext50_32x4d(alpha: float = 0.2, beta: float = 1.0, activation: str = 'relu', base_conv: ~torch.nn.modules.conv.Conv2d = <class 'nfnets.base.ScaledStdConv2d'>, **kwargs: ~typing.Any) → NFResNet

ResNeXt-50 32x4d model from “Aggregated Residual Transformation for Deep Neural Networks”. and “High-Performance Large-Scale Image Recognition Without Normalization” <https://arxiv.org/pdf/2102.06171v1>. :param pretrained: If True, returns a model pre-trained on ImageNet :type pretrained: bool

nfnets.models.resnet.nf_wide_resnet101_2(alpha: float = 0.2, beta: float = 1.0, activation: str = 'relu', base_conv: ~torch.nn.modules.conv.Conv2d = <class 'nfnets.base.ScaledStdConv2d'>, **kwargs: ~typing.Any) → NFResNet

Wide ResNet-101-2 model from “Wide Residual Networks”. and “High-Performance Large-Scale Image Recognition Without Normalization” <https://arxiv.org/pdf/2102.06171v1>. The model is the same as ResNet except for the bottleneck number of channels which is twice larger in every block. The number of channels in outer 1x1 convolutions is the same, e.g. last block in ResNet-50 has 2048-512-2048 channels, and in Wide ResNet-50-2 has 2048-1024-2048. :param pretrained: If True, returns a model pre-trained on ImageNet :type pretrained: bool

nfnets.models.resnet.nf_wide_resnet50_2(alpha: float = 0.2, beta: float = 1.0, activation: str = 'relu', base_conv: ~torch.nn.modules.conv.Conv2d = <class 'nfnets.base.ScaledStdConv2d'>, **kwargs: ~typing.Any) → NFResNet

Wide ResNet-50-2 model from “Wide Residual Networks”. and “High-Performance Large-Scale Image Recognition Without Normalization” <https://arxiv.org/pdf/2102.06171v1>. The model is the same as ResNet except for the bottleneck number of channels which is twice larger in every block. The number of channels in outer 1x1 convolutions is the same, e.g. last block in ResNet-50 has 2048-512-2048 channels, and in Wide ResNet-50-2 has 2048-1024-2048. :param pretrained: If True, returns a model pre-trained on ImageNet :type pretrained: bool