AI safety for carers: privacy, mistakes and consent

Privacy: what actually happens to what you type

A chat window feels private, like a conversation, but it is not the same as talking to a person. What you type is sent to a company's servers, may be stored, may be used to improve the model depending on the product and its settings, and may be reviewed by a human for safety or quality checks. Free tiers are generally more likely to use conversations for training than paid tiers, but settings and policies change, and "it's just a chat" is not a safe assumption. Treat anything you type as if it could be read by someone other than the assistant.

This does not mean these tools are unsafe to use. It means the responsibility for what goes into them sits with you, the way it would if dictating a letter to an assistant you had just met: useful for the task, not the place for anything that would cause harm if seen by the wrong person.

The redaction checklist

Before pasting a document or a chunk of notes into an assistant, remove:

• Full names, including family members and clinicians
• Dates of birth
• Home addresses
• Medical record, patient, Medicare or insurance numbers
• Bank details, card numbers and any financial account number
• Photos of documents showing any of the above; an assistant that reads images reads a name in a photo as easily as in typed text

A useful habit: type "I have removed names and identifying numbers" at the top of anything you paste. It costs nothing, and it forces a moment of checking that you actually have.

The specific never-list

Beyond the checklist above: never paste a complete, unredacted medical record; never paste a government ID number, including a Social Security, Medicare or passport number; never paste bank or card details, even to draft a letter about them; never paste a full name alongside a home address; never upload a photo of an ID card, Medicare card or bank statement. If a task seems to require one of these, it almost never does; there is nearly always a way to describe the situation without the identifying detail attached.

Hallucination: confident and wrong looks the same as confident and right

An assistant generates text that is plausible given the patterns it has seen, not text checked against a verified source for this situation. It reads exactly as confidently as a correct answer, which is what makes the error hard to spot.

Asked about a medication, it may invent a dose that sounds standard but is wrong for this person's age or kidney function. Asked about interactions, it may claim two drugs interact when they do not, or miss a real interaction between two less common ones. Asked about eligibility for a benefit, it may state a rule confidently that was true years ago, or true in a different country, or invented outright. Asked for an agency's phone number, it may produce one that looks plausible and is entirely wrong, because it is generating a number in the right format rather than looking one up.

None of this comes with a warning label. The fix is a habit: anything that would change a dose, a decision or an application gets checked against the label, the clinician, or the official website. Use an assistant to organise, draft and explain; use the primary source to confirm.

Consent and dignity

The person being cared for is a person, not a dataset. Where they are able to, they should know an assistant is being used to draft their care plan, organise their medical information or write on their behalf, and they should get a say in what is written and what is typed into a tool. A plan written entirely about someone, without their input, drifts toward what is convenient for the carer rather than what the person wants.

Where the person cannot give informed consent, for example due to advanced dementia or a medical crisis, use judgment: keep what you share to what the task needs, avoid anything that would embarrass or expose them if seen by someone else, and involve other family members in what gets written down. Dignity does not disappear because someone cannot advocate for it themselves.

What to do if something feels wrong

If an answer feels off, trust that instinct and check the primary source before acting. If you realise afterward that you pasted something you should not have, most assistants let you delete the conversation; do that, and be more careful next time. None of this is a reason to avoid these tools. It is the short list of habits that make them safe to use for something this personal.

I am about to paste a document into you. First, remind me in one short list what I should remove: names, dates of birth, addresses, ID and insurance numbers, and bank details. Wait for me to confirm before continuing.

Answering the next question, mark any part of your answer that is inference rather than something stated directly in the material I gave you, using the phrase 'this is inferred, not stated'. I have removed names and identifying numbers.

[paste your question or material here]

You just gave me an answer involving a dose, a rule, a number or a contact detail. List, briefly, exactly what I should independently verify before acting on it, and where, such as a label, a clinician, or an official website.

This is not medical, legal or financial advice.