Extended Optimizers in Variational Autoencoders Using Shrinkage Estimation

Mokhtari Farivar, Heydar; Nasiri, Parviz

doi:10.30473/jaslda.2026.77814.1000

- فلوچارت مراحل ورود و ثبت نام در سامانه نشریات علمی وزارت عتف

- راه اندازی سیستم مدیریت نشریات علمی دانشگاه پیام نور

تعداد نشریات	49
تعداد شماره‌ها	1,330
تعداد مقالات	11,467
تعداد مشاهده مقاله	24,803,055
تعداد دریافت فایل اصل مقاله	16,574,861

	Extended Optimizers in Variational Autoencoders Using Shrinkage Estimation
Journal of Advances in Statistical Learning and Data Analysis
مقالات آماده انتشار، پذیرفته شده، انتشار آنلاین از تاریخ 23 مهر 1405 اصل مقاله (2.73 M)
نوع مقاله: Original Article
شناسه دیجیتال (DOI): 10.30473/jaslda.2026.77814.1000
نویسندگان
Heydar Mokhtari Farivar^* ¹؛ Parviz Nasiri²
¹Iran University of Science and Technology (IUST), Tehran, Iran
²Department of Statistics, Payame Noor University , Tehran, Iran
چکیده
Abstract. This paper explores the implementation and comparison of various optimization algorithms for training Variational Autoencoders (VAEs). We introduce a shrinkage-based post-hoc weight-update rule and compare it against SGD, Adam, RMSprop, Adagrad, AdamW, and Nadam, using a multi-seed evaluation protocol (5–10 random seeds per configuration) on both a small synthetic dataset and real MNIST digits, together with a per-dimension KL-divergence diagnostic for posterior collapse. At each dataset’s default configuration, the shrinkage step appears to have a large and opposite effect: on synthetic data it induces complete posterior collapse in every trained model, while on MNIST it instead avoids the partial posterior collapse exhibited by standard adaptive optimizers, at a substantial cost to reconstruction quality. However, directly testing the robustness of the MNIST finding, by sweeping the learning rate and by testing a nonlinear architecture, shows that this apparent separation between collapsing and non-collapsing optimizers is not stable: SGD collapses at higher learning rates, the adaptive methods’ collapse is a non-monotonic function of learning rate that vanishes outside a narrow band, and two of the four shrinkage variants collapse completely under a nonlinear architecture. We conclude that posterior collapse in this setting depends on an interaction between optimizer, learning rate, and architecture that single-configuration comparisons cannot capture, and we report a fully reproducible implementation, including the robustness-check scripts, as a cautionary and diagnostic contribution rather than as a general-purpose optimizer recommendation.
تازه های تحقیق
Compares six standard optimizers with a shrinkage-based weight-update rule for VAE training. Evaluates posterior collapse using synthetic data and real MNIST digits. Introduces per-dimension KL-divergence diagnostics for posterior collapse. Demonstrates sensitivity to learning rate and neural network architecture. Provides reproducible implementations and robustness-check scripts for VAE optimization.
کلیدواژه‌ها
Keywords. Variational Autoencoders؛ Posterior Collapse؛ Shrinkage Estimation؛ Adaptive Optimizers؛ Reproducibility

مراجع
[1] Kingma, D. P., & Welling, M. (2013). Auto-Encoding Variational Bayes. arXiv preprint arXiv:1312.6114. https://doi.org/10.48550/arXiv.1312.6114 [2] Kingma, D. P., & Welling, M. (2019). An Introduction to Variational Autoencoders. Foundations and Trends in Machine Learning, 12(4):307–392. https://doi.org/10.1561/ 2200000056 [3] Rezende, D. J., & Mohamed, S. (2015). Variational Inference with Normalizing Flows. Proceedings of the 32nd International Conference on Machine Learning (ICML), PMLR 37:1530–1538. https://doi.org/10.48550/arXiv.1505.05770 [4] Robbins, H., & Monro, S. (1951). A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3), 400–407. https://doi.org/10.1214/aoms/1177729586 [5] Ruder, S. (2016). An Overview of Gradient Descent Optimization Algorithms. arXiv preprint arXiv:1609.04747. https://doi.org/10.48550/arXiv.1609.04747 [6] Rezende, D. J., Mohamed, S., & Wierstra, D. (2014). Stochastic Backpropagation and Approximate Inference in Deep Generative Models. Proceedings of the 31st International Conference on Machine Learning (ICML), PMLR 32(2):1278–1286. https://doi.org/10. 48550/arXiv.1401.4082 [7] Polyak, B. T. (1964). Some Methods of Speeding Up the Convergence of Iterative Methods. USSR Computational Mathematics and Mathematical Physics, 4(5), 1–17. https://doi. org/10.1016/0041-5553(64)90137-5 [8] Sutskever, I., Martens, J., Dahl, G. E., & Hinton, G. E. (2013). On the Importance of Initialization and Momentum in Deep Learning. Proceedings of the 30th International Conference on Machine Learning, PMLR 28(3):1139–1147. [9] Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980. https://doi.org/10.48550/arXiv.1412.6980 [10] Kingma, D. P., & Ba, J. (2015). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (ICLR), 1–13. https://doi.org/10. 48550/arXiv.1412.6980 [11] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. [12] Tipping, M. E., & Bishop, C. M. (1999). Probabilistic Principal Component Analysis. Journal of the Royal Statistical Society: Series B, 61(3):611–622. https://doi.org/10.1111/ 1467-9868.00196 [13] Tieleman, T., & Hinton, G. (2012). Lecture 6.5—RMSProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning. [14] Hinton, G. E. (2012). Neural Networks for Machine Learning. Coursera. [15] Higgins, I., Matthey, L., Pal, A., et al. (2017). beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. International Conference on Learning Representations (ICLR). [16] Doersch, C. (2016). Tutorial on Variational Autoencoders. arXiv preprint arXiv:1606.05908. https://doi.org/10.48550/arXiv.1606.05908 [17] Duchi, J. C., Hazan, E., & Singer, Y. (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12, 2121– 2159. [18] Duchi, J., & Singer, Y. (2009). Efficient Online and Batch Learning Using Forward Backward Splitting. Journal of Machine Learning Research, 10, 2899–2934. [19] Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., & Bengio, S. (2016). Generating Sentences from a Continuous Space. Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), 10–21. https://doi.org/10. 18653/v1/K16-1002 [20] Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., & Welling, M. (2016). Improved Variational Inference with Inverse Autoregressive Flow. Advances in Neural Information Processing Systems (NeurIPS), 29:4743–4751. https://doi.org/10.48550/arXiv. 1606.04934 [21] Rezende, D. J., & Viola, F. (2018). Taming VAEs. arXiv preprint arXiv:1810.00597. https: //doi.org/10.48550/arXiv.1810.00597 [22] Zhao, S., Song, J., & Ermon, S. (2017). InfoVAE: Information Maximizing Variational Autoencoders. arXiv preprint arXiv:1706.02262. https://doi.org/10.48550/arXiv.1706. 02262 [23] LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11):2278–2324. https://doi.org/ 10.1109/5.726791 [24] Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. International Conference on Learning Representations (ICLR). https://doi.org/10.48550/ arXiv.1711.05101 [25] Dozat, T. (2016). Incorporating Nesterov Momentum into Adam. ICLR Workshop Track.
آمار تعداد مشاهده مقاله: 6 تعداد دریافت فایل اصل مقاله: 2

سامانه مدیریت نشریات علمی. قدرت گرفته از سیناوب

پیوندهای مفید

اخبار و اعلانات

آمار

Extended Optimizers in Variational Autoencoders Using Shrinkage Estimation