Quantization and the number of LLM parameters in practice – testing local models for writing prose (Q8 or 30B?) – part I

How much can you really squeeze out of local LLMs when you want to keep everything private: no cloud, no powerful GPUs, and no “magic” context that remembers everything? Instead of theorizing, I ran a series of tests on a regular laptop, treating quantization, parameter count, and RAG access as the main “knobs” for manipulating the output. What I came up with wasn’t just a story, but also a set of surprisingly specific observations about when Q6/Q8 can outperform a larger model, why RAG tends to lose detail, and why a “good prompt” is sometimes just the beginning of problems.

Selection of equipment, tools and sources for testing LLMs

Sometimes you can’t or don’t want to use the cloud. Sometimes it’s better to run the model on local hardware – for various reasons. How much can you then “get” from such a model? Of course, the answers will vary depending on the application, and the results are difficult to assess in any other way than subjectively, but it’s worth trying, if only to better understand how this “new-not-new” technology like AI works.

From the outset, this test was intended to be as “home-based” as possible: I assumed realistic constraints (working locally, without sending data externally, without cloud-based models) and tested whether the entire creative process, not just individual text generations, could be based on them. I used “average” hardware as a reference point, meaning my used laptop (Ryzen 5 5500U and 32 GB RAM), so that the results would be repeatable for someone without a workstation with an expensive graphics card. I selected models from those publicly available on HuggingFace (with due regard for licenses and the sheer number of models), and I processed the entire process using a tool that allows me to work locally and test the approach with documents/RAGs. Because the local models were poor at handling Polish, I collected more sensible responses in English and then translated everything manually.

Moreover, I wanted as much of the process as possible to be “on the side” of the models, so I tried to delegate as much of the processes to the models as possible – without any of my own writing preparation, only using common sense and a methodical approach.

The “home” computer was my used laptop worth about 1000 PLN, so I really wanted to make this process accessible to the average user. The computer’s key specifications include an AMD Ryzen 5 5500U and 32GB of RAM (32GB is necessary to load models with parameters up to 30B—perhaps not with the best quantization, but sufficient for efficient operation). Sometimes, response generation was very slow, but that’s a price you have to accept given such limitations.

This test is, by definition, “non-technical” because it doesn’t measure what can be easily calculated and objectively verified (e.g., the accuracy of facts, solving math problems, or coding). It measures what is most important and simultaneously most subjective in prose writing: style, naturalness of language, the “density” of scenes, cause -and- effect coherence in a longer text, the credibility of dialogue, and whether characters have any depth or relationships that actually “play” with each other. Furthermore, there are a large number of “mathematical” tests online, while prose ones are rare, so I thought this was an interesting and somewhat different approach. In practice, this means that instead of benchmarks, what mattered to me was the quality of the material that could be transformed into a story (the timeline of events, character behavior, the consequences of characters’ decisions), so it was more of a creative and practical test than a hard-core engineering experiment.

The entire article below is “semi-technical”—in the sense that if you only want to use AI at a basic level, this won’t be for you. If you work with AI professionally or academically, this article isn’t for you either. However, if you’re looking for ways to improve your AI work, for example, through model selection, RAGs, or prompt engineering, you might find some interesting tips here.

The creative process and expected problems

From the beginning, I assumed that most of my time would not be spent on “writing,” but on the friction between my expectations and how local models actually behave in longer creative work.

I hadn’t published a short story or book yet, so the creative process was new to me; moreover, I always stuck to the constraint that models should be the authors of as much data as possible. Therefore, I knew I would need to complete at least two full “creative loops,” because during the first one, I would be learning about the problems I would be facing.

Before we can proceed, however, we need to describe the number of parameters in a model and what their quantization is. In short, “parameters” are the millions or billions of numerical weights that make up the trained model—the more of them (e.g., 7B vs. 30B), the greater its capacity for linguistic patterns, dependencies, and nuances, but also the more memory it requires to run locally and work meaningfully on longer text. In my test, this wasn’t an academic consideration, but a hard hardware limitation: on a “home” laptop, in practice, any decision about model size immediately impacts whether the model will fit in RAM. That’s where quantization comes in: it’s a technique for “averaging” the same parameters in a less precise notation (e.g., Q4/Q6/Q8), which allows the model to use less memory and usually run faster, but it comes at the cost of a potential loss of quality—sometimes subtle (style, vocabulary, rhythm), sometimes brutal (coherence, cause – and-effect logic, attention to detail). That’s why the “bigger model vs. better quantization” theme recurs so strongly throughout this article: intuition suggests that more parameters will always win, and in practice (especially when writing prose), it often matters whether the model has enough “precision” in its weightings to maintain the tone, relationships between characters, and consequences of events without drifting apart after a few paragraphs. And this is the context in which you should read the following sections: not as a guide to “which model is best,” but as a description of the trade-offs that immediately arise when you want to write locally, on ordinary hardware, and at the same time expect something more than a correct but flat story.

First creative loop – is the concept feasible?

https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard – was a huge help during the first “loop,” which resulted in two “sub-problems”:

The LLM leaderboard was the only reasonable point of reference, but it’s now archived. A second version was supposed to be created, but it hasn’t been created yet. You can read about the problems with leaderboards and why they’re unreliable here: https://huggingface.co/spaces/open-llm-leaderboard/blog – and yes, there are other leaderboards (for example, https://llm-stats.com/ ), but they suffer from the same problems as the Open LLM Leaderboard.

– There was also an additional “mini-problem”: no local LLM (at the time of writing this article, Bielik did not exist, but as a single model it would not provide a sufficient basis for the entire study) uses even a little bit of correct Polish. However, since I speak English well and received the best quality answers in that language, I simply decided to translate the whole thing by hand at the end of the creative process.

In this document, I use descriptions, prompts, and answers in both Polish and English as needed.

The second problem was choosing a local tool. For testing, I used GPT4All, which was simply perfect for this application. Above all, it proved very easy to use and has a built-in document management feature – the so-called RAG. You can read about choosing a desktop LLM client here: https://sailingbyte.com/blog/the-ultimate-comparison-of-free-desktop-tools-for-running-local-llms/  to better understand why I chose GPT4All.

The third problem was that I didn’t know what components I would need – hence, in the first loop, I asked a few LLMs about what components are needed for a good story and only then could I base my work on that.

The fourth problem turned out to be querying the LLMs appropriately to obtain the desired response. Suffice it to say that after executing over 100 queries, I began to understand how to formulate queries—this is, of course, what’s known as prompt engineering. It’s pointless to include all the prompts in this document, but I’ll cite some key ones here.

The fifth problem turned out to be that the language of the selected LLMs was rather poor, and the events were too simple. Of course, I could have simply revised the story by hand, but I knew I would achieve better quality if I started the process over.

The results after completing this loop were simply poor. But armed with a better understanding of the mechanics and an idea of how to work within these constraints, I approached the topic a second time, from the very beginning.

First loop – technical description of the test process

First model list

Models in GPT4All for the first stage I mainly chose popular models

Necessary components

Query for Phi3 and Qwen2.5:

I want to create a story. List what elements should I prepare to make a good story with properly described relations between characters and precisely placed in the world?

I have combined the responses into the following list:

1. The subject of the work – a description of the general setting of the world
2. Description the world presented
3. Safety data sheets characters
4. Literary style and recipient

I wanted the AI models to describe what was needed, and then to describe each of these elements. This was part of the plan, which, in my opinion, allowed for the creation of a coherent context for the depicted world. I decided that only after having a coherent, shared context would it be worthwhile to move forward and begin describing actual events.

With all this in mind, it was possible to move on to planning the course of events.

Choosing the topic and main thread

I considered unusual thematic combinations with references to AI and had 3 main ideas:

1. A small medieval town is plagued by theft. The sheriff attempts to use the “Justice System”—a mysterious entity that analyzes patterns—to uncover the perpetrators. He enlists the help of his assistant, who, however, distrusts the System.
2. In the Middle Ages, where a skilled blacksmith has managed to create AI-powered full-plate armor, war breaks out. The blacksmith’s assistant helps him hide and evade the tech hunters hunting him, but will he remain loyal to him until the end?
3. In a world where humans can dream up a separate life, Dream Moderators find work. They surf the semi-conscious brainwaves of sleeping people , monitoring for any aberrations. Navigation, however, is very difficult, and requires the assistance of highly sophisticated AI. But what if the AI decides to feed on the brainwaves it has access to or deliberately disrupts the Moderator’s navigation?

Overall, I chose the first theme, although the third one also tempts me greatly for future experiments, as it presents a potential for abstraction that could be enhanced with a bit of controlled model hallucination. The deciding factor in choosing the first theme was the fact that it allows for three styles (medieval, cowboy, and AI), and I simply loved that.

Description of the world presented

The second element was essential for further work. The Phi3 model was responsible for generating the context of the depicted world, and I expanded on these descriptions. Below is the translated version.

History of the world

In the charming medieval village of Eldridge, nestled between rolling hills and dense forests, life has always been simple. The people are hardworking and content with their lifestyle, but recent mysterious thefts are beginning to cast a shadow over this peaceful existence.

The first theft occurred two years ago, when a wealthy merchant’s horse was stolen from the town square during the noon market. Since then, similar incidents have plagued Eldridge: bags of gold vanishing in broad daylight, silverware vanishing without a trace, and even cattle being taken at night while their owners slept soundly. The townspeople are growing restless, wondering if this is just a passing phase or something more sinister.

Cultural context

Eldridge is known for its strong sense of community and tradition. People here value honesty above all else; it’s not uncommon for strangers to help each other with chores, and children learn the importance of fairness from a young age. The town has a rich history of crafts, particularly in carpentry and blacksmithing.

Mystical and technological elements

Although magic isn’t as common here, there are certain ancient artifacts that possess mystical power. One such artifact, known as the “Heart Stone,” can be found deep within the forest and is said to grant wisdom when spoken aloud. The city also boasts several skilled blacksmiths who have developed unique techniques for crafting weapons and armor.

Recent events that have influenced history

– First theft: Two years ago, during the midday market, the horse of a wealthy merchant was stolen. He complained to the king, and since he was a respected nobleman, the king decided to send his specialist to the area.

– The Heartstone Incident: A year ago, an elderly woman discovered the Heartstone in the forest. She spoke to it in a voice to gain wisdom, but ultimately became lost in the forest for three days before her son found her. The village knows the Stone’s location and benefits from its wisdom.

Geography and climate

Eldridge is a small town surrounded by lush green hills that stretch as far as the eye can see. The climate here is temperate, with warm summers and cool winters. In spring, wildflowers cover the fields in vibrant shades of yellow, purple, and pink.

• Town Square: A bustling hub where farmers sell their produce, bakers display fresh bread, and blacksmiths work with metal.
• Galdor Forest: To the east lies a vast expanse of ancient trees that whisper secrets to those who listen. The forest is home to many creatures, both familiar and strange.

Cities and neighborhoods

Eldridge itself has a population of around 200, but there are several smaller villages nearby:

• Hilltop Village: A collection of wooden cottages on a hill with panoramic views.
• River Town: Built along the banks of a winding river, where fishermen and boat people live.

Key residents

There are farmers toiling under the sun, blacksmiths whose hands move like dancers on hot coals, and merchants with carts full of goods from distant lands.

• Sheriff Jameson: A tall man with a strong jawline and deep-set eyes that seem to see right through you.
• Eleanor, the baker’s daughter: Petite and quick-witted, she has an infectious smile and can make any dish taste better than it should.
• Old Thorne: A seasoned veteran of many winters, with a worn face and full of stories.

Magical or technological infrastructure

The town is connected by wooden bridges that span the river. Blacksmiths have developed techniques for crafting weapons and armor, and bakers use ancient ovens to create breads as fresh as they are delicious. In Eldridge, technology isn’t advanced, but it is practical.

Unique features

• Heart Stone: A mystical artifact that supposedly grants wisdom when asked aloud.
• Eldridge Market: You can find everything from fresh eggs to handmade jewelry here.
• Old Thorne’s Inn: Known for solid meals and even more solid stories.

Character Data Sheets

I already had the key characters, refined them manually, and the Qwen model suggested a few more. I decided that one paragraph per character would be enough to start with. Overall, I ended up with the following:

• Sheriff Jameson: A focused and determined man with a refined sense of justice, Sheriff Jameson has a solid build and a chiseled jaw. His primary goal is to solve the Eldridge robberies while ensuring the safety of his community.
• Gideon: Logical and analytical, Gideon uses deduction to solve puzzles. He often taps his fingers when thinking deeply and tends to mutter to himself when working through problems.
• Eleanor: Bright and kind-hearted, Eleanor is a baker’s daughter who loves her cookies and enjoys experimenting with new recipes. She tends to hum softly while she works and values social connections above all else.
• Old Thorne: Sturdy and wise, Old Thorne tells stories about the history of Eldridge and crafts tools in his forge. He scratches his beard when he’s pondering something important and prefers wearing an old leather apron to any other clothing.
• Fuggus Thorne: Fuggus is a quick thinker with excellent observational skills, but he has trouble speaking due to a muteness stemming from a traumatic incident involving Gideon 20 years ago. He often uses hand gestures or pictures when he needs to communicate and wants to prove he’s an adult at Eldridge.
• Gynerva: Charismatic and a storyteller, Gynerva carries quirky, shiny objects to sell, wandering the markets with her deep auburn hair and green eyes that sparkle like autumn leaves. She twirls her hair excitedly when she talks and believes the world is full of wonders waiting for those who know where to look.
• Erica: Erica has a warm smile, charming manners, and strong leadership skills, but often appears shallow or selfish due to her tendency toward greed. Her rivalry with Sheriff Jameson drives many of her actions as she seeks greater respect from both the villagers and the king.

Literary style and audience

I wanted to start with something simple that LLMs could handle, so I simply manually defined the literary style as:

Aim for a well-read young adult audience, use rich, youthful language. Base your story on science fiction and fantasy styles, occasionally employing archaic vocabulary. Write as a third-person narrator.

The course of events

Having all the above elements, I could feed them into the model and ask it to create a story. The basic prompt, which I tested on Phi and Qwen, was :

You are a helper of professional writer. You are given in attachments elements of the world: topic description, world description, character cards and writing style. Your task is to generate list of events for a short story.

And this is where problems started to arise.

• Firstly: a trained eye will notice, that the elements from the previous subsections didn’t quite work together and sometimes contradicted each other—for example, the river motif, or the details surrounding the discovery and operation of the Heart Stone. However, during the development phase, this didn’t bother me or even bother me, so I continued with what I had.
• The prompt presented here caused the models to easily hallucinate—I don’t know why, until now. I suspect it had something to do with placing data directly in context, but…
• When I started placing world elements in RAG, the models used them in a very limited way, even leaving out most of the elements.

The result was that the initial suggestions were inconsistent and sometimes illogical. While the characters behaved quite well, the cause-and-effect relationships between the events were inconsistent. For example, one answer suggested that the Heart Stone was found once in the forest and then reappeared in the garden.

While the generated stories sounded interesting and could be manually improved to produce passable prose, that wasn’t the effect I was looking for. A second loop was therefore inevitable.

Summary of Part I

The first loop showed that simply collecting the elements of the world, the characters, and the main conflict is not enough for the model to create a coherent story. Although interesting ideas appeared and the characters were handled quite well, the whole thing began to fall apart at the level of event logic, cause-and-effect consistency, and the use of information from the RAG system. In practice, this meant hallucinations, contradictions, and scenes that sounded appealing but did not stick to the established assumptions.

That is why the second loop became necessary. In the next post, I will show what had to be changed to regain control over the story’s coherence and make the model work more consistently with the source material. I invite you to read the second part of the tests: