In this prototype, we’ve fine-tuned ChatGPT using oral history interview transcripts of World War II pilot Dick Nelms, a former B-17 Flying Fortress pilot. With natural language input, users can ask questions and receive audible responses from a 3D avatar, allowing them to explore Nelms’ training, bombing missions, combat experiences, and more firsthand knowledge.
The concept behind this project originally emerged from a customer support simulator experiment, where we integrated ChatGPT into Unity. This time, we’ve taken things to the next level by enabling users to communicate with the avatar in their own words. To make this possible, we leveraged a range of cutting-edge technologies:
- Initial input: Utilizing prompting engineering, we sent a prompt from Unity to OpenAI using their API that contained information about Dick Nelms based on transcripts from various interviews. https://openai.com/blog/openai-api
- Speech to Text: Wit.ai played a crucial role in transcribing natural language input into a format our AI could understand. https://wit.ai/
- Avatar Creation: For the lifelike representation of Dick Nelms, we turned to Ready Player Me. https://readyplayer.me/
- Text to Speech: Amazon Polly transformed our AI’s responses into audible narratives, bringing history to life. https://aws.amazon.com/es/polly/
- Face Animation (Lipsync): Oculus technology ensured that our avatar’s facial expressions and lipsync matched the spoken words. https://developer.oculus.com/documentation/unity/audio-ovrlipsync-unity/
- Integration: Unity served as the heart of our project, seamlessly bringing together these technologies to deliver a unique and immersive narrative exploration. https://unity.com
One of the main challenges we encountered was reducing the waiting times. Due to the integration of multiple technologies, the time it took to transfer information from one component to another was initially too long. However, with the right settings, we were able to create a smooth user experience.
In addition to this project, we have been working on a cross-media initiative centered around the universe of Dick Nelms. This initiative includes a website, a documentary film, a virtual world exhibit, and an AR experience. Now, we can add another layer to this multifaceted universe with AI technology.
Credit: Maria Sanchez Isaza | Paolo Tosolini
In collaboration with the NW EDU Network
Last Updated November 28, 2023