Activation functions in PyTorch (3)

Buy Me a Coffee☕

*Memos:

My post explains PReLU() and ELU().

My post explains SELU() and CELU().

My post explains Step function, Identity and ReLU.

My post explains Leaky ReLU, PReLU and FReLU.

My post explains GELU, Mish, SiLU and Softplus.

M…


This content originally appeared on DEV Community and was authored by Super Kai (Kazuya Ito)

Buy Me a Coffee☕

*Memos:

  • My post explains PReLU() and ELU().
  • My post explains SELU() and CELU().
  • My post explains Step function, Identity and ReLU.
  • My post explains Leaky ReLU, PReLU and FReLU.
  • My post explains GELU, Mish, SiLU and Softplus.
  • My post explains Tanh, Softsign, Sigmoid and Softmax.
  • My post explains Vanishing Gradient Problem, Exploding Gradient Problem and Dying ReLU Problem.

(1) ELU(Exponential Linear Unit):

  • can convert an input value(x) to the output value between aex - a and x: *Memos:
    • If x < 0, then aex - a while if 0 <= x, then x.
    • a is 1.0 by default basically.
  • is ELU() in PyTorch.
  • 's pros:
    • It normalizes negative input values.
    • The convergence with negative input values is stable.
    • It mitigates Vanishing Gradient Problem.
    • It mitigates Dying ReLU Problem. *0 is still produced for the input value 0 so Dying ReLU Problem is not completely avoided.
  • 's cons:
    • It's computationally expensive because of exponential operation.
    • It's non-differentiable at x = 0 if a is not 1.
  • 's graph in Desmos:

Image description

(2) SELU(Scaled Exponential Linear Unit):

  • can convert an input value(x) to the output value between λ(aex - a) and λx: *Memos:
    • If x < 0, then λ(aex - a) while if 0 <= x, then λx.
    • λ=1.0507009873554804934193349852946
    • α=1.6732632423543772848170429916717
  • is SELU() in PyTorch.
  • 's pros:
    • It normalises negative input values.
    • The convergence with negative input values is stable.
    • It mitigates Vanishing Gradient Problem.
    • It mitigates Dying ReLU Problem. *0 is still produced for the input value 0 so Dying ReLU Problem is not completely avoided.
  • 's cons:
    • It may cause Exploding Gradient Problem because a positive input value is increased by the multiplication with λ.
    • It's computationally expensive because of exponential operation.
    • It's non-differentiable at x = 0 if a is not 1.
  • 's graph in Desmos:

Image description

(3) CELU(Continuously Differentiable Exponential Linear Unit):

  • is improved ELU, being able to differentiate at x = 0 even if a is not 1.
  • can convert an input value(x) to the output value between aex/a - a and x: *Memos:
    • If x < 0, then aex/a - a while if 0 <= x, then x.
    • a is 1.0 by default basically.
  • 's formula is: Image description
  • is CELU() in PyTorch.
  • 's pros:
    • It normalises negative input values.
    • The convergence with negative input values is stable.
    • It mitigates Vanishing Gradient Problem.
    • It mitigates Dying ReLU Problem. *0 is still produced for the input value 0 so Dying ReLU Problem is not completely avoided.
  • 's cons:
    • It's computationally expensive because of exponential operation.
  • 's graph in Desmos:

Image description


This content originally appeared on DEV Community and was authored by Super Kai (Kazuya Ito)


Print Share Comment Cite Upload Translate Updates
APA

Super Kai (Kazuya Ito) | Sciencx (2024-10-05T19:01:26+00:00) Activation functions in PyTorch (3). Retrieved from https://www.scien.cx/2024/10/05/activation-functions-in-pytorch-3/

MLA
" » Activation functions in PyTorch (3)." Super Kai (Kazuya Ito) | Sciencx - Saturday October 5, 2024, https://www.scien.cx/2024/10/05/activation-functions-in-pytorch-3/
HARVARD
Super Kai (Kazuya Ito) | Sciencx Saturday October 5, 2024 » Activation functions in PyTorch (3)., viewed ,<https://www.scien.cx/2024/10/05/activation-functions-in-pytorch-3/>
VANCOUVER
Super Kai (Kazuya Ito) | Sciencx - » Activation functions in PyTorch (3). [Internet]. [Accessed ]. Available from: https://www.scien.cx/2024/10/05/activation-functions-in-pytorch-3/
CHICAGO
" » Activation functions in PyTorch (3)." Super Kai (Kazuya Ito) | Sciencx - Accessed . https://www.scien.cx/2024/10/05/activation-functions-in-pytorch-3/
IEEE
" » Activation functions in PyTorch (3)." Super Kai (Kazuya Ito) | Sciencx [Online]. Available: https://www.scien.cx/2024/10/05/activation-functions-in-pytorch-3/. [Accessed: ]
rf:citation
» Activation functions in PyTorch (3) | Super Kai (Kazuya Ito) | Sciencx | https://www.scien.cx/2024/10/05/activation-functions-in-pytorch-3/ |

Please log in to upload a file.



There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.

Related Posts