RWKV: reinventing RNNs for the transformer era
The Fifth Elephant paper reading meet-up - April 2024
-
Pinned update
Parking and security details for the papers’ reading session This update is for participants only
About the Paper
In the last three years, RNNs have caught up with the unparalleled capabilities of Transformers. The promise of Receptive Weighted Key Value (RWKV) is that this novel architecture combines the desirable aspects of both RNNs and Transformers: the massively parallelizable transformer-esque training, and RNN’s consistent computational and memory complexity during inference.
RWKV (pronounced as “RwaKuv”) is an attention-free language model, theoretically capable of handling an “infinite” context length.
Key takeaways for audience
- Intuitive understanding of RWKV’s formulation, using math and code.
- Information about performance benchmarks and the scaling laws - and evaluate how you can use RMKV in your work.
- A demo of RWKV’s inference prowess.
About the presenter
Yashodeep Deshmukh is Deputy Manager at Ashok Leyland.
RSVP
This paper discussion will be held at Atlassian’s office in EGL, Bangalore. In-person attendance is free. The Fifth Elephant members can join remotely to watch the live stream.
About The Fifth Elephant monthly paper discussions
The monthly discussions are organized to understand popular papers in Generative AI, DL, and ML domains. Papers are curated to benefit the community. The paper discussion is organized on the first Friday of each month, from 5:30 PM - 7:00 PM.
How you can contribute
- Suggest a paper at https://hasgeek.com/fifthelephant/call-for-papers/sub
- Moderate/discuss a paper someone else is proposing.
- Pick up a membership to support The Fifth Elephant’s activities.
- Spread the word among colleagues and friends. Join The Fifth Elephant Telegram group or WhatsApp group.
Contact
For inquiries, leave a comment or call The Fifth Elephant at +91-7676332020.
Hosted by
The Fifth Elephant - known as one of the best data science and Machine Learning conference in Asia - has transitioned into a year-round forum for conversations about data and ML engineering; data science in production; data security and privacy practices. more
Supported by
Venue host
Atlassian unleashes the potential of every team. Our agile & DevOps, IT service management and work management software helps teams organize, discuss, and complete shared work. The majority of the Fortune 500 and over 300,000 companies of all sizes worldwide - including NASA, Audi, Kiva, Deutsche B… more
-
Pinned update
Parking and security details for the papers’ reading session This update is for participants only
About the Paper
In the last three years, RNNs have caught up with the unparalleled capabilities of Transformers. The promise of Receptive Weighted Key Value (RWKV) is that this novel architecture combines the desirable aspects of both RNNs and Transformers: the massively parallelizable transformer-esque training, and RNN’s consistent computational and memory complexity during inference.
RWKV (pronounced as “RwaKuv”) is an attention-free language model, theoretically capable of handling an “infinite” context length.
Key takeaways for audience
- Intuitive understanding of RWKV’s formulation, using math and code.
- Information about performance benchmarks and the scaling laws - and evaluate how you can use RMKV in your work.
- A demo of RWKV’s inference prowess.
About the presenter
Yashodeep Deshmukh is Deputy Manager at Ashok Leyland.
RSVP
This paper discussion will be held at Atlassian’s office in EGL, Bangalore. In-person attendance is free. The Fifth Elephant members can join remotely to watch the live stream.
About The Fifth Elephant monthly paper discussions
The monthly discussions are organized to understand popular papers in Generative AI, DL, and ML domains. Papers are curated to benefit the community. The paper discussion is organized on the first Friday of each month, from 5:30 PM - 7:00 PM.
How you can contribute
- Suggest a paper at https://hasgeek.com/fifthelephant/call-for-papers/sub
- Moderate/discuss a paper someone else is proposing.
- Pick up a membership to support The Fifth Elephant’s activities.
- Spread the word among colleagues and friends. Join The Fifth Elephant Telegram group or WhatsApp group.
Contact
For inquiries, leave a comment or call The Fifth Elephant at +91-7676332020.
Venue
Atlassian
3rd Floor, Sunriver,
Wework Embassy Golf Links Business Park, Domlur
Bangalore - 560071
Karnataka, IN
Loading…
Hosted by
The Fifth Elephant - known as one of the best data science and Machine Learning conference in Asia - has transitioned into a year-round forum for conversations about data and ML engineering; data science in production; data security and privacy practices. more
Supported by
Venue host
Atlassian unleashes the potential of every team. Our agile & DevOps, IT service management and work management software helps teams organize, discuss, and complete shared work. The majority of the Fortune 500 and over 300,000 companies of all sizes worldwide - including NASA, Audi, Kiva, Deutsche B… more