It could be your charger has current limit, if decently designed.
If you use R you rely on charger design for safety and to prevent LED burnout.
And if charger puts out more V than you have LED current changing as well,
eg. not regulated.
If you use LM317 definitely need a heatsink on it...