The impact of using Large Language Models on the performance of Recommender Systems

Abstract

This thesis investigates the integration of Large Language Models (LLMs) into Conversational Recommender Systems (CRSs), evaluating the impact of model scale and hyperparameter configurations on retrieval performance. Moving beyond purely synthetic benchmarks, this research introduces a novel, end-to-end evaluation framework grounded in real-world user behaviour. To fuel this framework, a curated dataset of 734 query-target pairs, validated through a user study, was synthesised directly from historical clickstream and filtering logs from the price comparison platform Geizhals. Empirical evaluations using this framework reveal three primary insights. First, LLMs exhibit significant limitations when autonomously translating complex, multi-step user behaviours into single natural language intents, frequently suffering from hallucinations or context loss. Second, during active product retrieval via SQL generation, a model’s parameter scale is a strong predictor of systemic reliability and absolute hit rates; however, the smaller, code-specialised model (Qwen-Coder-30B) demonstrates superior precision and ranking quality due to more stringent filtering logic. Finally, a grid search analysis shows that hyperparameter tuning (Temperature and Top-p) exerts only a marginal impact on overall performance, showing that CRS success is driven predominantly by robust prompt architecture and autonomous fallback mechanisms rather than granular sampling adjustments.