Successful collaboration in computer-mediated teams requires awareness among group members of each other’s knowledge, skills, and goals. Large Language Models (LLMs) can play a mediating role in establishing and maintaining this awareness among group members. In an in-situ study, we explored the impact of an LLM-based chatbot on cognitive and social group awareness through a distributed text-based group task. We instructed participants (N = 48) to complete a travel-planning task in sixteen groups of three, with each member given conflicting goals. Each chat was complemented by a chatbot that could be asked for assistance. Through a survey and semi-structured interview, we gained insight into participants’ deliberations on the task and the chatbot’s role. We found that the chatbot’s presence helped increase group awareness as users are forced to clearly and transparently formulate their intentions when prompting the chatbot. The chatbot’s ability to provide suggestions that compromise between user goals based on the chat history helped participants reach a consensus. We present implications for the design of chatbots for collaborative settings.