Unreal Engine, Code Llama + LoRA
It’s exciting times for working with Large Language models for computer programming – especially for generalists like me who work across a range of platforms, languages and task-spaces that are often not very conversant with each other. Plus I am a ‘lazy’ self-taught programmer – there is far too much documentation out there in the docuverse to find and absorb for a single human being to get their head around – but once you ‘get’ how programs work, it’s often not too difficult to plod through to a solution.
To speed this up, I am very keen on the AI-to-the-rescue approach, even if the code produced is a bit buggy or wonky, because it very frequently provides a great foundation for me to work from and to figure out how all the pieces of the jigsaw puzzle might fit together. Plus the commenting and explanations provided by LLMs are also often illuminating. I’m not necessarily after elegant, efficient code (I’d leave that to some code genius somewhere) – just stuff that works well enough for me to get to where I want to go. It’s all about the ideas and working around the (sometimes hilarious and sometimes thought-provoking) LLM hallucinations.
I took this approach during my recent 6-month ANAT Playable Earth residency with Chris Fluke at the SmartSat CRC, Swinburne Centre for Astrophysics and Supercomputing, where ChatGPT 3 and 4 became invaluable companions in working out some approaches for working with Python, Google Colab and Unreal Engine 5 for data visualization of multi and hyperspectral satellite data. Prospectively combining this with deep learning approaches and in situ human interaction is an exciting prospect – add the capabilities of Unreal Engine, Cesium for Unreal, Omniverse and XR – well, so much to explore. I’ve covered this in some detail in my Playable Earth blog here and here.
Sometimes the best thing to do is to sit back for a while and let ideas percolate through the mind, which is what I’ve been doing for the last couple of months, whilst focusing on some other important things. But now it’s time to look again and see what’s developed over that time.
Of great interest is the oobabooga text-generation web UI, as it says on the box its “goal is to become the AUTOMATIC1111/stable-diffusion-webui of text generation.” Fantastic – this means that rather than relying on services like OpenAI ChatGPT, HuggingFace or Google Bard etc, it is possible for someone like me to start experimenting with locally installed LLMs that can run on my Nvidia RTX3090. I have a bunch installed already like GPT4ALL, privateGPT and others, and it’s been interesting to learn about LangChain and Vector Databases, BabyAGI and agents.
Agents are currently a step too far for what I want to do: I’m interested in IA (intelligence augmentation), not so much AI (artificial intelligence) – as part of my creative problem-solving toolset.
Which brings me to Unreal Engine. UE is huge and the documentation is huge too and sometimes it’s very tricky to find what you want and figure out how to solve a problem. So wouldn’t it be great to use an LLM to help? Well, yes, but ChatGPT’s knowledge cut-off in 2021 means it only covers up to UE4.26 and some announced features of UE5. BUT, with a local LLM of course one can train a local model through the magic of LoRA – an efficient and fairly rapid form of fine-tuning. Of course, I’m not alone in this thought and came across bublint’s ue5-llama-lora. Fantastic.
The upshot of this is that it should be possible to locally run a ‘programming’ LLM model like Meta’s recent codellama locally in the webUI, and fine-tune it using the LoRA approach and the bublint data, to create a fairly reliable UE5 programming oracle – exactly what I need!
I’ll update this page as I progress.
