Conversation

ruizewang

Description:
Add the YellowFin optimizer,called YFOptimizer.

YellowFin is an auto-tuning optimizer based on momentum SGD which requires no manual specification of learning rate and momentum. It measures the objective landscape on-the-fly and tunes momentum as well as learning rate using a local quadratic approximation.

We can use YFOptimizer just like Adam and SGD.
For example:
from fastNLP.core.optimizer import YFOptimizer

trainer = Trainer(model=model, n_epochs=100, optimizer=YFOptimizer(lr=0.01), train_data=train_data, dev_data=dev_data, loss=CrossEntropyLoss(), metrics=AccuracyMetric() )

paper: https://arxiv.org/abs/1706.03471
code: https://.com/JianGoForIt/YellowFin_Pytorch

Main reason: Add a fancy optimizer, which performs well in many tasks, though has its shortcomings. The biggest advantage is that you can use the default parameters directly, no need to adjust LR manually. (Note: there are also some people say they get a bad performance.)

Checklist 检查下面各项是否完成

Please feel free to remove inapplicable items for your PR.

  • The PR title starts with [$CATEGORY] (例如[bugfix]修复bug,[new]添加新功能,[test]修改测试,[rm]删除旧代码)
  • Changes are complete (i.e. I finished coding on this PR) 修改完成才提PR
  • All changes have test coverage 修改的部分顺利通过测试。对于fastnlp/fastnlp/的修改,测试代码必须提供在fastnlp/test/
  • Code is well-documented 注释写好,API文档会从注释中抽取
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change 修改导致例子或tutorial有变化,请找核心开发人员

Changes: 逐项描述修改的内容

  • 在optimizer.py中新引用了两个库math, numpy
  • 添加了一个优化器YellowFin optimizer (YFOptimizer).

Mention: 找人review你的PR

@修改过这个文件的人
@核心开发人员

Add the YellowFin optimizer,called YFOptimizer.

YellowFin is an auto-tuning optimizer based on momentum SGD which requires no manual specification of learning rate and momentum. It measures the objective landscape on-the-fly and tunes momentum as well as learning rate using a local quadratic approximation.

paper: https://arxiv.org/abs/1706.03471
code: https://.com/JianGoForIt/YellowFin_Pytorch
@codecov-io

Codecov Report

Merging #130 into master will decrease coverage by 2.01%.
The diff coverage is 11.25%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #130      +/-   ##
==========================================
- Coverage      68%   65.99%   -2.02%     
==========================================
  Files          90       90              
  Lines        6286     6517     +231     
==========================================
+ Hits         4275     4301      +26     
- Misses       2011     2216     +205
Impacted FilesCoverage Δ
fastNLP/core/optimizer.py19.14% <11.25%> (-72.86%)⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3fa95b6...d5e8d61. Read the comment docs.

@ruizewangruizewang changed the title Update optimizer.py [new] Add the YellowFin optimizer (YFOptimizer) Jan 19, 2019
Sign up for free to join this conversation on . Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.