Experience Replay

As you wander in the woods, the sun begins to sink below the horizon. You are suddenly filled with joy as you remember similar scenes from your past. Maybe you visited this place with your loved ones when you were young or recently. Being here triggers memories that are special to you. If this doesn’t resonate with you, think of a time when you saw someone’s eyes light up as they experienced something new for the first time. The mere thought of it makes you feel the same emotions. Emotions are essential to us. They shape our memories, which are fuzzy versions of our stories [1]. Even though they don’t match reality exactly, we tend to focus on the positive aspects of our past events. Even in difficult times, moments like this can happen.

Imagine if you could recreate the warm and joyful moments of your life. You might think that nothing can match the original experience. The first time is always dear to you. But what if you could relive these memories with a twist? What if you could change them slightly for better or worse? It would be like stepping into a picture while surrounding yourself with more senses and emotions. The best is you could share this with your loved ones too.

In case you are wondering how it might be possible, the answer lies in the scaling of generative AI that will empower experience replay. Consider that you have some bits and pieces of a memory that desired to be relived. You will not only be able to see it through your perspective but also from someone else’s point of view! As graphics technology like virtual and augmented reality advances, the idea will soon become a reality. We could recreate any moment we want from our memories and experience it in a new way. This project is more achievable than ever before. Even now, we have some tools that can make this happen. With GPT-4 as a multi-modal language and stable diffusion generating images using control-net, we are not far from using our life memories to create immersive scenes. The future is exciting.

The basic idea is simple: you provide some input as text and multimedia, describing a scene or a story from your memory. These inputs can be as detailed or as vague as you like. You might even add extra contexts, such as your mood, location, and time. The more context you provide, the more faithful the output will be to your original vision. This detailed input will be then fed into some version of GPT-N, which can generate coherent and realistic text based on any prompt. GPT-N analyzes your input and tries to predict a sequence of events encoded into a vector space, where each vector represents an event.

These vectors can then be passed into a 3D scene generator capable of transforming it into a 3D scene. The 3D scene generator might be pre-trained on a large database of images and videos, so it already knows how to create realistic and diverse environments and characters. The 3D scene generator will adjust the style and tone of the scene to match your context and inputs. The result will be a stunning 3D scene, which you can view from any angle and perspective. You will also be able to interact with the scene and modify it as you wish. You can add or remove elements, change the lighting or the weather, zoom in or out, pause, or rewind. You can also switch between different modes of presentation: first-person, third-person, cinematic, or abstract.

The best part of experience replay is that it does not require any real-time computation. You do not need to have a powerful computer or a VR headset to enjoy it. You can generate the scene whenever you have some spare time, and then replay it whenever you want. This offline idea will let you reflect on your memories in contrast to real-time, which will be impulsive.

Experience replay is not only a way of reliving your personal memories. It is also a way of exploring your imagination. You will be able to create scenes from your favorite movies, books, or songs. You can use it to visualize your dreams or your goals. You can use it to express your emotions or your opinions. You can use it to learn new things or to teach others.

Experience replay will be the future media of art and storytelling that anyone can master. It will blur the boundaries between reality and fiction (eternal return)[2], between self and others, and between past and future. It is a new way of experiencing life in all its richness and diversity.

I would like to thank my friends and colleagues for discussing the idea and helping with the draft.

References

[1] Brosch T, Scherer KR, Grandjean D, Sander D. The impact of emotion on perception, attention, memory, and decision-making. Swiss Med Wkly. 2013 May 14;143:w13786. doi: 10.4414/smw.2013.13786. PMID: 23740562.

[2] Anderson, R. Lanier (17 March 2017). “Friedrich Nietzsche”. The Stanford Encyclopedia of Philosophy.