AI assistants for school websites: a practical evaluation guide

A school website AI assistant can help a family find a calendar, locate an enrollment page, or identify the right office to contact. Its usefulness depends on the information it can access, the questions it can handle, and what happens when it cannot give a reliable answer.

Evaluate those behaviors before choosing a tool. The questions below are requirements to discuss and test with any provider; they are not claims that a particular product already meets them.

Define the first job clearly

Start with a small set of public-information tasks. Ask the school office which questions arrive repeatedly and which published pages already answer them.

Suitable pilot questions might include:

• Where can I find the school calendar?
• How do I arrange a campus visit?
• Which page explains the enrollment process?
• Who handles transportation questions?

Write down the expected source and next step for each question. Keep individualized decisions, private student records, and account changes outside a public-information pilot.

Generative AI can produce confident but inaccurate answers. NIST identifies this risk, along with privacy and human-oversight concerns, in its generative AI risk profile. NIST's Generative AI Profile.

The practical implication is to test behavior, including failure cases, rather than judging only how naturally the assistant writes.

Control which sources it uses

Ask the provider to explain how approved school content enters the system, how updates are detected, and how an incorrect source can be removed.

For each pilot topic, name the authoritative page and its staff owner. If the current enrollment page conflicts with last year's handbook, resolve the content conflict before expecting the assistant to resolve it reliably.

During a demonstration, ask for the source of an answer and follow the link. Check whether the page actually supports the answer, whether it belongs to the correct campus, and whether it is current. A link beside an answer is useful only when the linked content backs up what was said.

Test updates too. Change a date in a test environment, then ask the provider to demonstrate when the assistant stops giving the previous date.

Test uncertainty and staff handoffs

The assistant needs a useful response when information is missing or a question requires school staff.

A helpful fallback might say: “I couldn't confirm that on the school's published pages. Contact the enrollment office through this page.” Use a verified destination. Do not invent office hours or promise a response time the school has not agreed to provide.

Check that families can reach that handoff without repeatedly rephrasing the question. A visible contact option should remain available throughout the interaction.

Use a review sheet like this during the pilot:

These tests cannot prove that an assistant will never fail. They give the school a repeatable way to find problems and decide whether the pilot is ready to expand.

Ask specific questions about conversation data

Families may enter personal information even if the assistant does not ask for it. Review that possibility with the school's privacy and technology teams.

Ask where conversations are stored, who can access them, how long they are retained, whether they are used to train models, and what deletion and export controls exist. Check the answers against the proposed contract and actual configuration.

Use synthetic questions and fictional data during evaluation. Avoid placing student records or real family conversations into a demo simply to see what happens.

Explain the assistant's purpose in the interface and direct account-specific requests to the school's approved process. Have the appropriate school reviewer determine the legal and contractual requirements; a marketing label is not a substitute for that review.

Check the interface with the same care as a form

Test opening and closing the assistant, typing a question, reading a response, following a link, and contacting staff with a keyboard and relevant assistive technology. Check focus behavior, control labels, and how newly arrived messages are announced.

Ask reviewers to try the panel on a small screen and with enlarged text. Include supported languages in the test plan. W3C's form guidance provides a useful starting point for controls and feedback, alongside a broader accessibility evaluation.

Review the pilot and keep an owner

Our recommended pilot log includes the question, expected source, answer quality, whether the next step worked, and the correction owner. Keep sensitive conversation content out of routine reports unless the school's approved process specifically requires it.

Track source accuracy, unresolved questions, and successful staff handoffs. Review recurring unanswered questions as possible website content gaps. Raw conversation volume does not show whether families found useful information.

Agree on who can pause the assistant if it begins giving incorrect or inappropriate responses. Retest important questions after changes to source content, model settings, or connected services.

Bring your own questions to the demonstration

Choose five questions families really ask and two situations the assistant should hand to staff. Ask the provider to demonstrate both, then document what still needs work.

Explore the EdPilotHub AI assistant offering, or contact our team to discuss the scope and evaluation requirements for your school.