[D] LLMs for classification task

Hey folks, in my project we are solving a classification problem. We have a document , another text file (consider it like a case and law book) and we need to classify it as relevant or not.

We created our prompt as a set of rules. We reached an accuracy of 75% on the labelled dataset (we have 50000 rows of labelled dataset).

Now the leadership wants the accuracy to be 85% for it to be released. My team lead (who I don’t think has high quality ML experience but says things like do it, i know how things work i have been doing it for long) asked me to manually change text for the rules. (Like re organise the sentence, break the sentence into 2 parts and write more details). Although i was against this but i still did it. Even my TL tried himself. But obviously no improvement. (The reason is because there is inconsistency in labels for dataset and the rows contradict themselves).

But in one of my attempts i ran few iterations of small beam search/genetic algorithm type of thing on rules tuning and it improved the accuracy by 2% to 77%.

So now my claim is that the manual text changing by just asking LLM like “improve my prompt for this small dataset” won’t give much better results. Our only hope is that we clean our dataset or we try some advanced algorithms for prompt tuning. But my lead and manager is against this approach because according to them “Proper prompt writing can solve everything”.

What’s your take on this?