{"id":460,"date":"2021-04-13T14:58:00","date_gmt":"2021-04-13T05:58:00","guid":{"rendered":"https:\/\/arithmer.blog\/?p=460"},"modified":"2022-03-08T15:44:28","modified_gmt":"2022-03-08T06:44:28","slug":"transformer-introduction","status":"publish","type":"post","link":"https:\/\/arithmer.blog\/blog\/transformer-introduction","title":{"rendered":"Transformer\u7d39\u4ecb"},"content":{"rendered":"\n<p class=\"has-small-font-size\">\u672c\u8cc7\u6599\u306f2021\u5e7404\u670813\u65e5\u306b\u793e\u5185\u5171\u6709\u8cc7\u6599\u3068\u3057\u3066\u5c55\u958b\u3057\u3066\u3044\u305f\u3082\u306e\u3092WEB\u30da\u30fc\u30b8\u5411\u3051\u306b\u30ea\u30cb\u30e5\u30fc\u30a2\u30eb\u3057\u305f\u5185\u5bb9\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u76ee\u6b21\"><strong>\u25a0\u76ee\u6b21<\/strong><\/h3>\n\n\n\n<p style=\"font-size:16px\">\u30fbTransformer\u306e\u6982\u8981<br>\u30fbTransformer\u306e\u5404\u69cb\u9020\u89e3\u8aac<\/p>\n\n\n\n<ol style=\"font-size:16px\"><li>Positional Encoding<\/li><li>Attention<\/li><li>Multi-headed Scaled Dot-Product Self-Attetion<\/li><li>Shortcut Connection, Layer Normalization, Position-wise Feedforward Network<\/li><li>Transformer\u306e\u5b9f\u9a13\u7d50\u679c\u30fb\u8003\u5bdf<\/li><\/ol>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u6982\u8981-transformer\u3063\u3066\u4f55\"><strong>\u25a0[\u6982\u8981] Transformer\u3063\u3066\u4f55\uff1f<\/strong><\/h3>\n\n\n\n<p style=\"font-size:16px\">2017\u5e74\u306e\u8ad6\u6587Attention Is All You Need[1]\u3067\u767a\u8868\u3055\u308c\u305f\u30e2\u30c7\u30eb<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"297\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_01.jpg\" alt=\"\" class=\"wp-image-490\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_01.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_01-300x87.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_01-768x223.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_01-304x88.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<p class=\"has-text-align-center\" style=\"font-size:16px\">\u6a5f\u68b0\u7ffb\u8a33\u30bf\u30b9\u30af\u306b\u304a\u3044\u3066\u65e2\u5b58SOTA\u3088\u308a\u3082\u9ad8\u3044\u30b9\u30b3\u30a2\u3092\u8a18\u9332<\/p>\n\n\n\n<p class=\"has-text-align-center has-small-font-size\">[1] Vaswani, Ashish et al. \u201cAttention is All you Need.\u201d ArXiv abs\/1706.03762 (2017)<\/p>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u6982\u8981-transformer\u306e\u4ed6\u5206\u91ce\u3078\u306e\u5fdc\u7528\"><strong>\u25a0[\u6982\u8981] Transformer\u306e\u4ed6\u5206\u91ce\u3078\u306e\u5fdc\u7528<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"445\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_02.jpg\" alt=\"\" class=\"wp-image-491\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_02.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_02-300x130.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_02-768x334.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_02-304x132.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<p class=\"has-text-align-center has-small-font-size\">[2] Dosovitskiy, A. et al. \u201cAn Image is Worth 16\u00d716 Words: Transformers for Image Recognition at Scale.\u201d ArXiv abs\/2010.11929 (2020)<br>[3] Gulati, Anmol et al. \u201cConformer: Convolution-augmented Transformer for Speech Recognition.\u201d ArXiv abs\/2005.08100 (2020)<\/p>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u6982\u8981-nlp\u306b\u304a\u3051\u308b\u30e2\u30c7\u30eb\u306e\u767a\u5c55\"><strong>\u25a0[\u6982\u8981] NLP\u306b\u304a\u3051\u308b\u30e2\u30c7\u30eb\u306e\u767a\u5c55<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"446\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_03.jpg\" alt=\"\" class=\"wp-image-492\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_03.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_03-300x131.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_03-768x335.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_03-304x132.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u6982\u8981-rnn-cnn-vs-transfomer\"><strong>\u25a0[\u6982\u8981] RNN, CNN vs Transfomer<\/strong><\/h3>\n\n\n\n<p style=\"font-size:18px\"><strong>Recurrent Neural Network (RNN)<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"102\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_04.jpg\" alt=\"\" class=\"wp-image-493\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_04.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_04-300x30.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_04-768x77.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_04-304x30.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<p style=\"font-size:18px\"><strong>Convolutional Neural Network (CNN)<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"102\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_05.jpg\" alt=\"\" class=\"wp-image-494\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_05.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_05-300x30.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_05-768x77.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_05-304x30.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<p id=\"transformer\" style=\"font-size:18px\"><strong>Transformer<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"125\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_06.jpg\" alt=\"\" class=\"wp-image-495\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_06.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_06-300x37.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_06-768x94.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_06-304x37.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-transformer\u306e\u5168\u4f53\u56f3\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Transformer\u306e\u5168\u4f53\u56f3<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"483\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_07.jpg\" alt=\"\" class=\"wp-image-496\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_07.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_07-300x142.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_07-768x362.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_07-304x143.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-positional-encoding-\uff11\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac\u3011Positional Encoding(\uff11)<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"466\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_08.jpg\" alt=\"\" class=\"wp-image-497\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_08.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_08-300x137.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_08-768x350.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_08-304x138.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-positional-encoding-\uff12\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Positional Encoding(\uff12)<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"466\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_09.jpg\" alt=\"\" class=\"wp-image-498\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_09.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_09-300x137.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_09-768x350.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_09-304x138.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-positional-encoding-\uff13\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Positional Encoding(\uff13)<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"479\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_10.jpg\" alt=\"\" class=\"wp-image-499\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_10.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_10-300x140.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_10-768x359.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_10-304x142.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<p class=\"has-text-align-center has-small-font-size\">[4] http:\/\/jalammar.github.io\/illustrated-transformer \u3088\u308a<\/p>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-attention\u6a5f\u69cb-\uff11\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Attention\u6a5f\u69cb(\uff11)<\/strong><\/h3>\n\n\n\n<p style=\"font-size:16px\">\u5f93\u6765\u306eSeq2Seq\u578b\u306eRNN\u7ffb\u8a33\u30e2\u30c7\u30eb<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"366\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_11.jpg\" alt=\"\" class=\"wp-image-500\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_11.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_11-300x107.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_11-768x275.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_11-304x109.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-attention\u6a5f\u69cb-\uff12\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Attention\u6a5f\u69cb(\uff12)<\/strong><\/h3>\n\n\n\n<p style=\"font-size:16px\">Attention\u3092\u5229\u7528\u3057\u305fSeq2Seq\u578b\u306eRNN\u7ffb\u8a33\u30e2\u30c7\u30eb<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"389\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_12.jpg\" alt=\"\" class=\"wp-image-501\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_12.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_12-300x114.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_12-768x292.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_12-304x115.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-attention\u6a5f\u69cb-\uff13\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Attention\u6a5f\u69cb(\uff13)<\/strong><\/h3>\n\n\n\n<p style=\"font-size:16px\">Attention\u8a08\u7b97\u5f0f(dot product attention)<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"435\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_13.jpg\" alt=\"\" class=\"wp-image-467\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_13.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_13-300x127.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_13-768x326.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_13-304x129.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-attention\u6a5f\u69cb-\uff14\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Attention\u6a5f\u69cb(\uff14)<\/strong><\/h3>\n\n\n\n<p style=\"font-size:16px\">Attention\u3092\u5229\u7528\u3057\u305fSeq2Seq\u578b\u306eRNN\u7ffb\u8a33\u30e2\u30c7\u30eb<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"387\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_14.jpg\" alt=\"\" class=\"wp-image-468\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_14.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_14-300x113.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_14-768x290.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_14-304x115.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-transformer\u306eattention\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Transformer\u306eAttention<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"465\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_15.jpg\" alt=\"\" class=\"wp-image-469\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_15.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_15-300x136.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_15-768x349.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_15-304x138.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-self-attention\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Self -Attention<\/strong><\/h3>\n\n\n\n<p style=\"font-size:16px\">Query, Key, Value\u304c\u5168\u3066\u540c\u3058Attention<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"397\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_16.jpg\" alt=\"\" class=\"wp-image-470\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_16.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_16-300x116.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_16-768x298.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_16-304x118.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-scaled-dot-product-attention\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Scaled Dot-Product Attention<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"422\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_17.jpg\" alt=\"\" class=\"wp-image-471\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_17.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_17-300x124.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_17-768x317.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_17-304x125.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-multi-head-attention-\uff11\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Multi -Head Attention(\uff11)<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"449\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_18.jpg\" alt=\"\" class=\"wp-image-472\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_18.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_18-300x132.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_18-768x337.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_18-304x133.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-multi-head-attention-\uff12\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Multi -Head Attention(\uff12)<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"460\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_19.jpg\" alt=\"\" class=\"wp-image-473\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_19.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_19-300x135.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_19-768x345.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_19-304x137.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-multi-head-attention-\uff13\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Multi -Head Attention(\uff13)<\/strong><\/h3>\n\n\n\n<p style=\"font-size:16px\">\u7dda\u5f62\u5199\u50cf\u3067\u884c\u5217\u306e\u6b21\u5143\u3092\u6e1b\u3089\u3057\u3066\u304b\u3089\u4e26\u5217\u5316(\u56f3\u306e\u4f8b\u3067\u306f3\u4e26\u5217)<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"469\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_20.jpg\" alt=\"\" class=\"wp-image-474\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_20.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_20-300x137.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_20-768x352.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_20-304x139.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-multi-head-attention-\uff14\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Multi -Head Attention(\uff14)<\/strong><\/h3>\n\n\n\n<p style=\"font-size:18px\"><strong>\u6570\u5f0f<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"94\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_21.jpg\" alt=\"\" class=\"wp-image-475\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_21.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_21-300x28.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_21-768x71.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_21-304x28.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<p style=\"font-size:18px\"><strong>\u306a\u305cMulti-Head\u306a\u306e\u304b\uff1f<\/strong><\/p>\n\n\n\n<ul style=\"font-size:16px\"><li>Attention\u3059\u308b\u7a7a\u9593\u3092\u9078\u629e\u3067\u304d\u3001\u9650\u5b9a\u3055\u308c\u305f\u7279\u5fb4\u91cf\u7a7a\u9593\u3067Attention\u3067\u304d\u308b<\/li><li>\u5358\u4e00Attention\u3067\u306f\u5e73\u5747\u7684\u306aAttention\u3068\u306a\u308b\uff08\u5927\u96d1\u628a\u306aAttention\uff09<\/li><li>\u305d\u3057\u3066\uff0c\u5358\u8a9e\u306e\u8981\u7d20\u306f\uff11\u3064\u3067\u306f\u306a\u3044\u306e\u3067Multi\u304c\u671b\u307e\u3057\u3044\uff08\u5358\u8a9e\u610f\u5473\uff0c\u54c1\u8a5e\uff09<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"124\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_22.jpg\" alt=\"\" class=\"wp-image-476\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_22.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_22-300x36.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_22-768x93.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_22-304x37.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-shortcut-connection\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Shortcut Connection<\/strong><\/h3>\n\n\n\n<p style=\"font-size:16px\">2015\u5e74\u306b\u63d0\u6848\u3055\u308c\u305fResidual Network(ResNet)[4]\u306e\u8981\u7d20<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"410\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_23.jpg\" alt=\"\" class=\"wp-image-477\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_23.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_23-300x120.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_23-768x308.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_23-304x122.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<p class=\"has-text-align-center has-small-font-size\">[4] He, Kaiming et al. \u201cDeep Residual Learning for Image Recognition.\u201d 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016).<\/p>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-layer-normalization\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Layer Normalization<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"458\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_24.jpg\" alt=\"\" class=\"wp-image-478\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_24.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_24-300x134.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_24-768x344.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_24-304x136.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-position-wise-feed-forward-networks\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Position-wise Feed-Forward Networks<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"468\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_25.jpg\" alt=\"\" class=\"wp-image-479\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_25.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_25-300x137.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_25-768x351.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_25-304x139.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-transformer-module\u306e\u307e\u3068\u3081\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Transformer Module\u306e\u307e\u3068\u3081<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"453\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_26.jpg\" alt=\"\" class=\"wp-image-480\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_26.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_26-300x133.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_26-768x340.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_26-304x134.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-transformer\u306e\u5168\u4f53\u56f3-\u518d\u63b2\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Transformer\u306e\u5168\u4f53\u56f3(\u518d\u63b2)<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"484\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_27.jpg\" alt=\"\" class=\"wp-image-481\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_27.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_27-300x142.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_27-768x363.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_27-304x144.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-rnn\u7cfb\u306e\u63a8\u8ad6\u65b9\u6cd5\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] RNN\u7cfb\u306e\u63a8\u8ad6\u65b9\u6cd5<\/strong><\/h3>\n\n\n\n<p style=\"font-size:16px\">\u5f93\u6765\u306eSeq2Seq\u578b\u306eRNN\u7ffb\u8a33\u30e2\u30c7\u30eb<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"175\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_28.jpg\" alt=\"\" class=\"wp-image-482\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_28.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_28-300x51.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_28-768x131.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_28-304x52.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<p class=\"has-text-align-center\" style=\"font-size:16px\">\u51fa\u529b\u3055\u308c\u305f\u3082\u306e\u3092\u30c7\u30b3\u30fc\u30c0\u306e\u5165\u529b\u306b\u518d\u5ea6\u5165\u529b\u3057\u3066\u5f97\u3066\u3044\u304f<\/p>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-transformer-decoder\u306e\u63a8\u8ad6-\u30a4\u30e1\u30fc\u30b8\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Transformer Decoder\u306e\u63a8\u8ad6(\u30a4\u30e1\u30fc\u30b8)<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"424\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_29.jpg\" alt=\"\" class=\"wp-image-483\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_29.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_29-300x124.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_29-768x318.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_29-304x126.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-transformer-decoder\u306e\u5b66\u7fd2\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Transformer Decoder\u306e\u5b66\u7fd2<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"442\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_30.jpg\" alt=\"\" class=\"wp-image-484\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_30.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_30-300x129.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_30-768x332.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_30-304x131.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u69cb\u9020\u89e3\u8aac-masked-attention\"><strong>\u25a0[\u69cb\u9020\u89e3\u8aac] Masked Attention<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"431\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_31.jpg\" alt=\"\" class=\"wp-image-485\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_31.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_31-300x126.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_31-768x323.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_31-304x128.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u5b9f\u9a13\u7d50\u679c-\u6a5f\u68b0\u7ffb\u8a33\u30bf\u30b9\u30af\u306e\u5b9f\u9a13\u7d50\u679c\"><strong>\u25a0[\u5b9f\u9a13\u7d50\u679c] \u6a5f\u68b0\u7ffb\u8a33\u30bf\u30b9\u30af\u306e\u5b9f\u9a13\u7d50\u679c<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"336\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_32.jpg\" alt=\"\" class=\"wp-image-486\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_32.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_32-300x98.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_32-768x252.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_32-304x100.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<p class=\"has-text-align-center\" style=\"font-size:16px\">\u65e2\u5b58\u306e\u624b\u6cd5\u3088\u308a\u3082\u9ad8\u3044\u30b9\u30b3\u30a2 \uff06 \u5b66\u7fd2\u30b3\u30b9\u30c8\u3082<\/p>\n\n\n\n<p class=\"has-text-align-center has-small-font-size\">[1] Vaswani, Ashish et al. \u201cAttention is All you Need.\u201d ArXiv abs\/1706.03762 (2017)<\/p>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u5b9f\u9a13\u7d50\u679c-self-attention\u306e\u8a08\u7b97\u91cf\"><strong>\u25a0[\u5b9f\u9a13\u7d50\u679c] Self -Attention\u306e\u8a08\u7b97\u91cf<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"270\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_33.jpg\" alt=\"\" class=\"wp-image-487\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_33.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_33-300x79.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_33-768x203.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_33-304x80.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<ol style=\"font-size:16px\"><li>\ud835\udc5b &lt; \ud835\udc51\u306e\u5834\u5408\u306f\uff0cSelf-Attention\u306e\u8a08\u7b97\u91cf\u306fRNN\u3088\u308a\u3082\u5c0f\u3055\u304f\u306a\u308b<\/li><li>RNN\u306f\u524d\u306e\u5024\u3092\u5f85\u3064\u5fc5\u8981\u304c\u3042\u308b\u306e\u3067\u8a08\u7b97\u6642\u9593\u306f\u3088\u308a\u304b\u304b\u308b<\/li><li>Self-Attention\u306f\u9577\u671f\u306e\u4f9d\u5b58\u95a2\u4fc2\u3092\u5b66\u7fd2\u3057\u3084\u3059\u3044<\/li><\/ol>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u5b9f\u9a13\u7d50\u679c-\u5404\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u7cbe\u5ea6\u3078\u306e\u5f71\u97ff\"><strong>\u25a0[\u5b9f\u9a13\u7d50\u679c] \u5404\u30d1\u30e9\u30e1\u30fc\u30bf\u306e\u7cbe\u5ea6\u3078\u306e\u5f71\u97ff<\/strong><\/h3>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"1024\" height=\"484\" src=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_34.jpg\" alt=\"\" class=\"wp-image-488\" srcset=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_34.jpg 1024w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_34-300x142.jpg 300w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_34-768x363.jpg 768w, https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/NS20210413_34-304x144.jpg 304w\" sizes=\"(max-width: 1024px) 100vw, 1024px\"><\/figure>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"transformer\u307e\u3068\u3081\"><strong>\u25a0Transformer\u307e\u3068\u3081<\/strong><\/h3>\n\n\n\n<p style=\"font-size:16px\">1.Transformer\u306fAttention\u30d9\u30fc\u30b9\u69cb\u9020\u3067\u8a08\u7b97\u91cf\u30fbGPU\u89aa\u548c\u6027\u304c\u9ad8\u3044\u30e2\u30c7\u30eb<br>2.\u6642\u7cfb\u5217\u30c7\u30fc\u30bf\u3092\u4e00\u5ea6\u306b\u51e6\u7406\u3059\u308b\u305f\u3081Positional Encoding\u3092\u5b9f\u65bd\u3059\u308b\u5fc5\u8981<br>3.Transformer\u306eAttention\u306f\u4ee5\u4e0b\u306e\u8981\u7d20\u3092\u6301\u3064<\/p>\n\n\n\n<ul style=\"font-size:16px\"><li>Self-Attention\uff08\u5165\u529b\u540c\u58eb\u3092\u7528\u3044\u3066\u7279\u5fb4\u91cf\u3092\u518d\u69cb\u7bc9\uff09<\/li><li>Scaled Dot-Product Attention\uff08\u52fe\u914d\u60c5\u5831\u3092\u4fdd\u6301\u3057\u3064\u3064\u5185\u7a4d\u3059\u308b\uff09<\/li><li>Multi-Headed Attention\uff08\u8907\u6570\u306e\u30d8\u30c3\u30c9\u3092\u7528\u3044\u3066\u4e26\u5217\u306bAttention\uff09<\/li><\/ul>\n\n\n\n<p style=\"font-size:16px\">4.\u30c7\u30b3\u30fc\u30c0\u3067\u306f\u672a\u6765\u306e\u60c5\u5831\u306e\u30ea\u30fc\u30af\u3092\u9632\u3050\u305f\u3081\u306bMasked Attention\u3059\u308b<br>5.\u5b9f\u9a13\u7d50\u679c\u304b\u3089Transformer\u306e\u9ad8\u6027\u80fd\uff0c\u4f4e\u8a08\u7b97\u91cf\u304c\u793a\u3055\u308c\u305f<\/p>\n\n\n\n<h3 class=\"has-medium-font-size wp-block-heading\" id=\"\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9\"><strong>\u25a0\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9<\/strong><\/h3>\n\n\n\n<p style=\"font-size:16px\"><a href=\"https:\/\/arithmer.blog\/wp-content\/uploads\/2022\/02\/19_Transformer\u7d39\u4ecb.pdf\" target=\"_blank\" rel=\"noreferrer noopener\">Transformer\u7d39\u4ecb.pdf<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u672c\u8cc7\u6599\u306f2021\u5e7404\u670813\u65e5\u306b\u793e\u5185\u5171\u6709\u8cc7\u6599\u3068\u3057\u3066\u5c55\u958b\u3057\u3066\u3044\u305f\u3082\u306e\u3092WEB\u30da\u30fc\u30b8\u5411\u3051\u306b\u30ea\u30cb\u30e5\u30fc\u30a2\u30eb\u3057\u305f\u5185\u5bb9\u306b\u306a\u308a\u307e\u3059\u3002 \u25a0\u76ee\u6b21 \u30fbTransformer\u306e\u6982\u8981\u30fbTransformer\u306e\u5404\u69cb\u9020\u89e3\u8aac Positional E &#8230; <\/p>\n","protected":false},"author":3,"featured_media":489,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[20,51,28,50,35,24,52,36],"_links":{"self":[{"href":"https:\/\/arithmer.blog\/index.php?rest_route=\/wp\/v2\/posts\/460"}],"collection":[{"href":"https:\/\/arithmer.blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/arithmer.blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/arithmer.blog\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/arithmer.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=460"}],"version-history":[{"count":8,"href":"https:\/\/arithmer.blog\/index.php?rest_route=\/wp\/v2\/posts\/460\/revisions"}],"predecessor-version":[{"id":731,"href":"https:\/\/arithmer.blog\/index.php?rest_route=\/wp\/v2\/posts\/460\/revisions\/731"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/arithmer.blog\/index.php?rest_route=\/wp\/v2\/media\/489"}],"wp:attachment":[{"href":"https:\/\/arithmer.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=460"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/arithmer.blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=460"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/arithmer.blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=460"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}