AI Chatbots in Healthcare: What Doctors Need to Know About HIPAA and Patient Privacy

Why Are Healthcare Practices Using AI Chatbots?

AI chatbots are becoming another way for medical practices to communicate with people outside traditional office hours. A chatbot can potentially:

• Answer questions about office hours and locations
• Explain how to schedule an appointment
• Provide general information about services
• Direct visitors to forms or resources
• Answer basic questions about insurance or billing
• Collect information for an appointment request
• Assist with intake
• Communicate with existing patients through a patient portal

A chatbot can handle routine questions without requiring staff to respond individually to every website visitor. However, the more information and responsibility a chatbot takes on, the more carefully the practice should evaluate the technology.

Can an AI Chatbot Collect Patient Information?

It can, but the practice needs to understand what happens when it does. A visitor might begin a conversation by asking about scheduling and then provide information such as “I’ve been diagnosed with depression and am looking for a therapist who treats it.”

The information may be sensitive health information even though the person has not yet become a patient. The practice should therefore determine what information the chatbot is designed to collect and whether visitors are likely to provide information beyond what the practice intended.

Before implementing a chatbot, ask:

• Does it collect names or contact information?
• Can users enter free-form messages?
• Does it collect symptoms or medical history?
• Does it save conversations?
• Where are conversations stored?
• Who can access them?
• How long are they retained?
• Can the vendor use conversations to train or improve its AI system?

What Should You Ask an AI Chatbot Vendor?

A chatbot vendor may advertise its platform as secure or HIPAA compliant. That information can be useful, but it should not be the end of the practice’s review. Before signing an agreement, ask:

• What information does the chatbot collect? Understand whether visitors can enter free-form text, upload documents, or provide health information.
• Where is the information stored? Find out where conversations are stored, how long they are retained, and whether the practice can delete them.
• Who can access the information? Determine whether the vendor, its employees, subcontractors, or other third parties can access chatbot conversations.
• Is the information used to train AI models? This is an especially important question for generative AI tools. The practice should understand whether information submitted through the chatbot may be used for model training, product development, analytics, or other purposes.
• Does the vendor provide a Business Associate Agreement (BAA)? If the chatbot will handle protected health information (PHI) on behalf of the practice, determine whether the vendor is a business associate and whether an appropriate BAA is available.
• What happens if the chatbot is wrong? The contract and implementation plan should account for inaccurate or inappropriate responses, including how problems are reported and addressed.

The practice should review the vendor’s actual agreements rather than relying solely on statements made on its website.

Can an AI Chatbot Give Medical Advice?

Answer this question before implementing a healthcare chatbot. A chatbot can be designed to provide general information. That is very different from allowing it to diagnose a condition, recommend treatment, interpret symptoms, or tell someone whether they need immediate medical attention.

Generative AI can produce confident-sounding answers even when the underlying information is incomplete or incorrect.

That does not mean every healthcare chatbot should be prohibited from answering health-related questions. It means the practice should establish clear limits on what the chatbot is permitted to do. Those limits might include:

• Providing general educational information
• Directing visitors to emergency services when appropriate
• Recommending that visitors contact the practice
• Refusing to diagnose conditions
• Avoiding treatment recommendations
• Transferring certain questions to a human

The chatbot’s capabilities should match the level of oversight the practice can realistically provide.

Should Patients Know They Are Talking to AI?

A practice should not lead visitors to believe that they are communicating with a physician, nurse, receptionist, or other human employee when they are actually communicating with an AI system.

A chatbot can clearly identify itself as an automated system and explain what it can and cannot do. For example, a practice might explain that the chatbot can answer general questions and assist with scheduling but is not a substitute for professional medical advice.

The practice should also consider whether users are given an opportunity to contact a human when the chatbot cannot appropriately address their question. Clear expectations can reduce confusion and make it less likely that someone treats an automated response as personalized medical guidance.

What About AI Chatbots and Patient Intake?

Some practices may want a chatbot to collect information before an appointment or begin the intake process. That raises additional privacy and compliance considerations.

If a chatbot collects medical history, insurance information, treatment history, or other similar information, the practice should determine how that information is protected and whether the chatbot’s technology and vendor relationship are appropriate for handling it.

In many cases, a secure patient portal or established intake system may be more appropriate than asking visitors to enter sensitive information into a general website chat window.

What About Chatbots for Mental Health Practices?

Mental and behavioral health practices should be particularly thoughtful about chatbot design because visitors may disclose highly sensitive information during an initial conversation.

A person looking for therapy might provide details about a diagnosis, trauma, family situation, medications, or current symptoms before ever scheduling an appointment. The practice should therefore understand what information the chatbot collects and how it is handled.

Illinois mental and behavioral health practices also need to consider state-specific requirements governing the use of AI in mental health care.

Can AI Chatbots Respond to Patient Reviews or Messages?

Chatbots and other AI tools can also be used to draft responses to patient communications and for other healthcare marketing purposes. This can be convenient, but practices should be careful about what information they put into the AI system.

For example, a practice might paste a patient’s online review into an AI tool and ask it to draft a response. If the practice adds information about the patient’s diagnosis or treatment to make the response more personalized, it may have introduced PHI into a system that was not approved to receive it.

Our team can advise you on HIPAA compliance for healthcare AI, including evaluating AI tools that may interact with PHI. Contact us today to book a free consultation if you have questions.

Create an AI Chatbot Policy Before Launch

A practice should establish clear rules before turning an AI chatbot loose on its website. The policy or implementation plan should address:

• Purpose: Define what the chatbot is intended to do and what it is not intended to do.
• Information collection: Identify what information the chatbot may collect and what information it should not request.
• Medical advice: Establish clear limits on diagnosis, treatment recommendations, and other clinical responses.
• Human escalation: Determine when a conversation should be transferred to a staff member or another appropriate resource.
• Privacy and security: Identify how chatbot conversations are stored, accessed, and protected.
• Vendor management: Establish who is responsible for reviewing the vendor’s contract, security practices, and HIPAA obligations.
• Monitoring: Determine how the practice will identify inaccurate, inappropriate, or potentially harmful chatbot responses.

AI systems can change as vendors update their models. A chatbot that behaves appropriately when it launches may produce different responses after an update. Ongoing review should therefore be part of the practice’s AI strategy.

A Pre-Launch Checklist for Healthcare Chatbots

Before adding an AI chatbot to your practice website, ask:

1. What is the chatbot allowed to do? Define its purpose and establish clear limits.
2. What information can visitors enter? Consider whether users are likely to disclose PHI or other sensitive information.
3. Where does that information go? Understand storage, retention, access, and data-sharing practices.
4. Is the vendor a business associate? If PHI is involved, determine whether HIPAA requires a BAA.
5. Does the vendor use conversations to train its AI? Review the vendor’s data-use provisions before implementation.
6. Can the chatbot provide medical advice? Set boundaries around diagnosis, treatment recommendations, and urgent health concerns.
7. When does a human need to step in? Create an escalation process for questions the chatbot should not handle.
8. Has the chatbot been tested? Test realistic questions, including sensitive and potentially urgent scenarios, before launch.
9. How will the practice monitor it? Establish a process for reviewing and correcting problematic responses.
10. Does state law impose additional requirements? Consider state privacy, healthcare, professional, and AI-specific laws that may apply.

A chatbot should be treated as part of the practice’s overall technology and compliance framework, not simply as another website feature.

Using AI Chatbots Without Losing Human Oversight

AI chatbots can make a practice more accessible and reduce the burden of answering routine questions. But the more information a chatbot collects and the more sophisticated its responses become, the more the practice needs to understand the legal and practical risks before implementation.

The safest approach is not necessarily to avoid AI. It is to define the chatbot’s purpose, limit what it can collect and say, understand the vendor relationship, and establish appropriate human oversight before patients begin using it.

If you are considering an AI chatbot for your practice, Jackson LLP’s healthcare attorneys can review the proposed technology, vendor agreement, and compliance considerations before you launch it.

Free Attorney Consultation

Book Now

Frequently Asked Questions About AI Chatbots in Healthcare