By Daniel McCloskey, Acuity Technical Manager 

An Opportunity for Hands-On LLM Learning 

The LLM League was structured to give practitioners of any level of experience an opportunity to interact with a Large Language Model. This was fortunate for me; my only experience with Large Language Models before the LLM League was in using them to review code for my projects and answer basic questions. I did have some experience with reinforcement learning after participating in earlier AWS DeepRacer events, but this was my first exposure to fine-tuning LLMs. 

The day’s events included a 2-hour workshop, personal time for fine-tuning, and a final head-to-head competition. The workshop was comprehensive and gave me the information I needed to kick off my first fine-tuning job.  

First Steps with LLM Fine-Tuning 

I started by using all default settings. I then prepared customized data for additional training jobs while the first one was running. I found out the hard way that fine-tuning works better with more data, so I combined the provided data with my generated data for my final training job during the workshop. To my surprise, that combination ended up performing the best of all my tests.  

It quickly became apparent that even with hours of dedicated time, we were only scratching the surface with the possibilities as I fine-tuned the data I had generated. For example, I was exposed to hyperparameters, which control how well the model learns, but we had limited time to tweak them.  

We finished the LLM League activities with the finale, a rapid-fire prompt engineering exercise. The competition itself was quick, and it got me thinking critically about how I phrase my prompts to make sure that the LLM response was accurate, relevant, and concise. The fast pace of the final round made for a nice contrast after running back-to-back training jobs during the workshop and fine-tuning portions of the event. 

Inspiration for Innovation 

The LLM League gave the Acuity team new insight into how LLMs can be used and fine-tuned. This will have a direct impact on how we serve our clients: federal agencies with targeted missions to serve and protect our nation’s citizens, global reputation, and critical assets. We gained hands-on experience that we can use when using AI tools across our focus areas of digital evolution, data enablement, and hyperautomation. My team already has a few ideas for integrating custom LLMs into our projects, so the workshop was useful for learning how to implement them. We’re planning to use existing data from a project to tailor models to more accurately anticipate what users expect and need from the system. 

Acuity’s AWS Journey 

The LLM League, and all the programming at the AWS Public Sector Summit, showed the AWS commitment to supporting and furthering the development of AI. As an AWS partner, Acuity is embracing the opportunity to further our AWS capabilities, which include: 

• DevSecOps Culture and Automation: Using services like CodePipeline, SAM, CloudFormation, and CDK to enable automation and continuous integration/continuous deployment (CI/CD) practices.  

• Containerization: Leveraging ECS, EKS, and ECR for container management and orchestration. 

• IT Modernization and Cloud Migration: Using services such as DMS, Glue, and EC2 to modernize IT infrastructure and migrate to the cloud. 

• Application Development: Developing applications with Java/Spring, React, Angular, API Gateway, Lambda, and CloudFront. 

• Cost Optimization: Implementing cost-saving measures with Autoscaling, TrustedAdvisor, Compute Savings Plans, and Reserved Instances  

• Cloud Architecture: Designing cloud architecture with VPC and the Well-Architected Framework. 

• Security: Ensuring security with services like Security Hub, Organizations, CloudTrail, Trusted Advisor, Secrets Manager, STS, and IAM.  

• Big Data: Managing big data with OpenSearch, DynamoDB, and S3. 

Advancing Together with LLM and AWS 

The AWS LLM League provided invaluable hands-on experience with Large Language Models, which will significantly enhance our AI capabilities—reach out today to discuss how we can put these skills to work for you.