Will AI Replace Your Doctor? Replicants and "Robo-proxies" with Dr. Jacob Appel

Mark Mullen: [00:00:00] Welcome back to the podcast, Dr. Appel. And this time I will ask you to introduce yourself to our audience. 

Jacob M. Appel, MD: Sure. I'm Jacob M. Appel. I'm a psychiatrist at the Icahn School of Medicine at Mount Sinai, but to my mother, I'm still her son who's not the rabbi. 

Mark Mullen: Okay, we'll leave it at that. Okay, first I'd like to follow up on our previous discussion about your emerging model to decisional capacity assessment, your values-based approach.

And in the values-based approach, unlike the Four Abilities Model by Applebaum and Grisso, the physician's task is to understand the patient's values and then determine whether or not the decision that they are making is consistent with those underlying values. So the physician is really just making one ruling, is what you're doing now consistent with the values that you're espousing and your longitudinal [00:01:00] values, as opposed to four rulings.

Do you understand, appreciate, communicate a clear choice, and, reason rationally, right? So in some ways it's a simpler model. The physician only has to make one decision. When I sat with it, I realized that the one decision that we're asking the physician to make is, who is this person as a human? What are their underlying values?

And I feel like that's a much more difficult determination to make than do they meet these simple cognitive criteria. Am I understanding your model correctly? 

Jacob M. Appel, MD: More or less, yes. So I would say is you're describing the first and most important step. Obviously, if the patient's choice doesn't match their underlying values, then you need to do more analysis and probably use elements of the Applebaum and Grisso model.

But in most cases where their values do match, it's not gonna be an issue, and that's what I think is the heart of capacity, which we're mostly talking about. A-and yes, I think your description of it is completely accurate. I, wouldn't say necessarily it involves more work. It involves a different kind of work.

Mark Mullen: [00:02:00] Yes, exactly. 

Jacob M. Appel, MD: Yeah. And probably a more meaningful kind of work because the information you'll gather may actually be beneficial and therapeutic to the patient, will build a connection, unlike the four skills model, which actually is somewhat disruptive to the therapeutic relationship. 

Mark Mullen: That has certainly been my longitudinal experience.

So then the next question is, i-in the American healthcare system in its current form, do you think that we are ready to actually execute a model like this? 

Jacob M. Appel, MD: I do. I think one thing that I love about being both a, an emergency and a consultation psychiatrist is it's the only area in psychiatry now where you're still not charging by the minute, and so I can spend a long time with a patient when necessary to come up with the right decision.

Mark Mullen: And I guess your, call is even if that's uncomfortable in some other settings, practice models, professions, we're going to need to put on our big boy pants because this is the patient's autonomy that we're talking about, so we may have to take a little extra time. We may have to step outside of traditional algorithms and simple yes/no questions and really try to understand the patient on that human [00:03:00] level, but you think that should be doable?

Jacob M. Appel, MD: Yes. I mean, nobody would ever say that a surgeon should do a procedure badly and quickly. So why in psychiatry do we not use the best approach we have, even if it takes more time and care? 

Mark Mullen: Fair enough. Okay, moving into our conversation for today, we're gonna talk about ethical considerations as AI becomes more powerful and pervasive.

And you've written about the possibility that artificial intelligence might become capable of replacing physicians, at least in some instances. How urgent of a concern is this for you? How close do you think AI is to replacing physicians in some contexts? 

Jacob M. Appel, MD: Well, I will offer my baseline thought that I am not one of these people who is opposed to using artificial intelligence in medicine.

I think it's gonna have a lot of advantages over time. I do think it's oversold in terms of how fast people think it's gonna develop to do the things we need it to do. As I remind people, AI right now is great for things that need to be right about 90 to 95% of the time. It's not so good for things that need to be right 100% of the time.

And unfortunately, medicine is one of those [00:04:00] areas where the stakes are very high and you really do need to get it right, unlike, let's say, writing a business letter, which you can go back and correct. So in that context, I think there we're already seeing pushes to, for example, with online therapies done by chatbots to sort of silo out at least some aspects of, medical and psychiatric care to AI before it's really ready to be rolled out in that way, and I think that's part of the problem.

Mark Mullen: And so we're gonna have this sort of philosophical discussion about what it might look like for AI to replace physicians. What are you imagining? So you mentioned like therapy chatbots. That would be one area. I could picture maybe taking that a step further and saying the patient's looking into a camera, and so the AI can do a physical exam visually, and maybe instead of a chatbot, the AI actually is a dynamic voice, which is more and more common now.

what are you picturing as sort of the logical conclusion if AI were to replace physicians in the future? 

Jacob M. Appel, MD: Well, my fear is that for high-end consumers, probably like you, [00:05:00] certainly like me, who can afford to pay for a doctor, we will go to a, a professional in an office. We'll go to a psychiatrist who's a human being, sit down on a couch and talk to them.

But for people who are covered by Medicare or Medicaid or poor insurance plans, they will come and they will talk to a machine that will scan their body, that will have a, quote-unquote, conversation with them, and that will offer a diagnosis and prescribe a treatment. And whether or not there's even a physician overlooking that won't be so clear.

There may not be. A- and I think there are major downsides to that beyond simply getting the medical decision-making accurate. Even if your robot doctor, your what I call a replicant doctor, were perfect at making diagnoses and treatment decisions, that would still not be sufficient to replace what human beings do.

When the robot says to you, "I'm really sorry about the loss of your mother," that means nothing. When you ask a robot, "If you were my sister, what would you do in these circumstances?" it can't give you meaningful advice, and that's a major problem. There, there is a value [00:06:00] in a human connection with your provider.

I like to tell the anecdote, my grandmother, when she would go to a doctor, the only two things she cared about were, one, whether he listened to her heart and lungs, and two, whether or not he asked about her grandchildren. She could be going to a podiatrist, and those were her priorities. If he did that, he was a good doctor.

If he didn't, he was a bad doctor. 

Mark Mullen: What are some reasons-- You know, there's some authenticity concerns there. There's some the replicant doesn't share human values or the human experience, and so its ability to put itself in your shoes and... I mean, I, love the question of what would you do if this were a family member?

That's something that I-- That's certainly the way that I try to make a lot of my medical decisions when I'm stuck. Makes sense that AI can't have a-- I don't, See, I don't even know if I can say this, but I guess it makes sense to me that AI can't have a sister. What are some areas, though, where you think humans might reasonably prefer a replicant, that is an AI doctor, to a human doctor?

Jacob M. Appel, MD: So I think the most obvious area is gonna be in Matters like diagnostic [00:07:00] assessment where we show you the EKG or the radio scan and you-- the machine is going to be better than a human being. We're already seeing that in detecting occult lung tumors, for example. The problem is we're still not-- we don't have a legal system designed to handle that yet.

And I've written about this as well. It's something called the negative outcome penalty paradox The problem is that if physicians use AI and the AI statistically is better than physicians at doing certain tasks under the current legal regime, if the doctor thinks the AI is wrong and overrules it, but it actually turns out the doctor's wrong, a jury is gonna find the doctor liable by saying, "Well, the machine is ninety-nine point nine percent accurate, and humans are only ninety-seven percent accurate."

On the other hand, if you don't overrule the machine and the machine turns out to be wrong, which will happen a certain small percentage of the time, the jury will say, "You're a human being. People go to the doctors to get your expert opinion. Why did you rely on the machine?" So either way, we increase the risk of liability, [00:08:00] and there might be ways of addressing that, but the legal system is not caught up with that yet, which is why if I were a patient, I'd still be somewhat reluctant, and certainly if I were a doctor, I'd be very reluctant to use these tools without very careful oversight.

Mark Mullen: You have no ability to hold these tools accountable if they mess up. Right. They're completely... Okay, that makes sense. 

Jacob M. Appel, MD: And there's also the something called the learned intermediary doctrine, which basically means the doctors, if they use tools like an EKG machine or a stethoscope, are responsible for understanding how it works and if it's doing things right or wrong.

I can do that with a stethoscope, even as a psychiatrist. I can't do that with an AI machine. 

Mark Mullen: Probably even if you were an AI expert, you would have a lot of difficulty. I mean, you-- Yeah, there's-- It's almost impossible to determine where the limit is of the AI's competence. 

Jacob M. Appel, MD: Yeah, it's a black box, so even the people who designed it don't know how it comes to make decisions in a granular way.

So nobody can sit down with a patient and say, "This is why the machine is deciding this." 

Mark Mullen: So in some cases, you may want a replicant because of its [00:09:00] superior ability to make decisions based on probability and perhaps have a higher certainty of diagnosis, be able to incorporate various sources of data more robustly.

Of course, it can scan the whole internet for information about diagnosing something that maybe a physician has had very limited, if not zero exposure to. What about the opposite? What are some features of a human doctor that you think a patient might prefer over a replicant? 

Jacob M. Appel, MD: Well, the first one that, that seems to be lost in a lot of people is common sense.

And I, understand AI has advanced a lot in the last year or two, but a couple of years ago, a colleague and I programmed a bunch of capacity scenarios into AI, and it actually did pretty well in doing them until we gave them a, patient who had vital signs of a heart rate of zero and blood pressure of zero over zero.

And it seemed to think the patient had capacity, and that's a problem. It did not seem to recognize the underlying medical emergency. So it's really good at dealing with what you ask it, but it doesn't pick up- things that [00:10:00] common sense would indicate to a human being. So that's one concern. Second concern is that machines are entirely without empathy.

And I know there's Paul Bloom's work on whether empathy is a good thing or a bad thing, and there may be some downsides to over-empathizing with patients in terms of equity and treating people unfairly. But even in that context, we, do want providers that care to some degree. we screen for them at medical school to make sure people aren't sociopaths when they come to medical school.

All machines, in essence, are sociopaths. They are, incapable of, moral decision-making in a, for lack of a better word, humanistic way 

Mark Mullen: I love that take. In the paper that I used when outlining this podcast, one of your quotes is, you're discussing authenticity and empathy from a human versus a replicant, and you say, "Such emotional support is not merely an amenity.

Rather, authentic emotional support is often an essential aspect of healing and recovery." Could you unpack this for me a little bit? 

Jacob M. Appel, MD: Sure. So there's a small percentage of people who go to doctors to be cured with a capital C. [00:11:00] Most people go to the doctors because they want help or healing, and the reality is, most of the conditions we see in people aren't curable.

They show up once, like you give them an antibiotic, and they walk away. They have chronic conditions that have social implications, that have deep psychological implications, whether they're physical or psychiatric ailments, and they want someone to escort them on that journey, so to speak. They want someone to be their companion in suffering and their companion in recovery, and machines can't do that in the same way because machines have never suffered and machines have never recovered.

Mark Mullen: And is this exclusive only to psychiatry, or do you feel like that principle applies across all the, the whole house of medicine? 

Jacob M. Appel, MD: Oh, I think that applies across the whole house of medicine. I mean, even the things that seem, like, really basic, like setting a bone or prescribing an antibiotic, people come back, and they don't talk about whether the treatment worked.

They talk about the doctor's bedside manner. 

Mark Mullen: That's so true. 

Jacob M. Appel, MD: And yeah. 

Mark Mullen: Do you think simulated empathy could be enough, or do you think that has to be actually real carbon-based human empathy? [00:12:00] 

Jacob M. Appel, MD: I'm not sure w- that sympathetic empathy is even a, a thing. Sympathetic, sympathy might be. Simulated sympathy might be a thing.

But, I mean, if a machine came up to you at your grandmother's funeral and said, "I'm really sorry about your loss," would that mean anything to you? I don't think to most people that would mean anything. It would mean that somebody programmed that machine to say that. 

Mark Mullen: Correct. Another thing you discuss is this concept of shared vulnerability, and you say that human physicians have to wield this authentic emotional presence rather than merely recognize patterns and pretend to be empathetic.

And I can't agree more with this. In fact, as a clerkship director, on the first day of the clerkship, I try to emphasize the importance of being a human in the room with a human. I give the example of a patient talking about how they wonder if it'd be better if they just weren't alive and an M3 who's on their clerkship might say, "Have you ever heard voices that other people don't?"

You know? It's like if you're not actually fully present and listening to the person and demonstrating that you're there and you care about them, then the patient's probably gonna walk out of the room, and they probably should. You're an ED psychiatrist. What does it mean for you to [00:13:00] demonstrate shared vulnerability in your clinical work?

Jacob M. Appel, MD: Sure. Well, I'll first mention that there's the old example of the other mistake that medical students often make is the patient will tell this horrific story of, suffering that is an extreme magnitude and not that common, and the patient- or the student will say to them like, "Oh, I know exactly what you're going through," which they clearly don't.

But-- And what obviously we are trained as psychiatrists is to be like, "I can't even imagine what you're going through." But implicit in that is, "But I'm trying to imagine what you're going through," and that's part of the connection. The machine can't be trying to imagine what you're going through. So I think in the emergency room, sometimes it's simply your very presence, the presence of another human being makes a huge difference, and I can't explain why having a present human being makes a difference and a machine doesn't.

Maybe it's evolutionary. Maybe I lef- let off pheromones. I don't know. But I think just presence and silence matters. And sometimes it's simply the simple courtesies of asking people, "Can I get you something to drink? Is there any way I can help you get home?" that makes their experience a little bit [00:14:00] less traumatic.

The machines can't-- They can do it in a formulaic way, but they can't do it with meaning. 

Mark Mullen: Yeah, and there's something about as a patient knowing this human, this physician is gonna take five minutes out of their day and go track down a sandwich for me that I think gives me inherent value, whereas a machine doing that is it doesn't mean anything to me.

I totally agree with that. 

Jacob M. Appel, MD: A-and it's also it's like you also know that at the other end of the line, so to speak, there's someone who you can at some level reason with or negotiate with or explain your story. I always think about like old, what is it, the Mike Nichols a-and, Elaine May routine where she plays the indifferent telephone operator who no matter how much she tries to explain to her, she simply says, "Please deposit another nickel."

the machine is like that. it-- no matter how good your reason, no matter what your explanation for why you did something, it simply has an algorithm. 

Mark Mullen: Your appeal to pathos means nothing 

Jacob M. Appel, MD: to the 

Mark Mullen: machine. 

Jacob M. Appel, MD: Nothing. Nothing at all You explain that I did this because, my wife was in labor. It will look up the algorithm, wife in labor, and decide whether or not that's a reason.

Mark Mullen: Let's discuss the concept of trust. So humans come with [00:15:00] all sorts of messy ulterior motives, and I think in my humble opinion, the most central thing connecting all humans is our own fallibility. And I think it goes without saying that a patient needs to trust their physician, whether it's a human or a replicant, and the patient needs to feel like the physician has their best interest at heart and is doing their best to help them get better.

How might the trust between a patient and physician change if the physician were human versus replicant? 

Jacob M. Appel, MD: Sure. So by definition, replicants are incapable of honesty, trustworthiness. They, do exactly what they're programmed to do and have all the glitches. So in some ways, the most sociopathic of individuals, I don't wanna diagnose, but I don't know, Attila the Hun is, I'm sure I, didn't violate the Goldwater rule too much. I was gonna say, 

Mark Mullen: call Goldwater. Okay. 

Jacob M. Appel, MD: Yeah, That, that is, is more sympathetic and more capable of trust than a machine by definition. It is the floor. Because there's nothing that you can be sure of that the machine won't release all of your private secrets to the [00:16:00] entire dark web in case there's some glitch.

You can be pretty confident that most doctors go into medicine to help people, and there are limits to what most doctors will do in terms of evil. Most of us are here to try to do good. 

Mark Mullen: That's been my experience. I don't know, though. I think there's a very significant portion of our culture that doesn't believe that physicians went into medicine to help people.

Jacob M. Appel, MD: Well, it's one of those things where people say they may not trust medicine in general, but they trust their own doctors, and you can build that trust pretty darn quickly, I have found, by being human, even in an emergency room. It's amazing how patients are there for 45 minutes, and already somehow you become their doctor sometimes.

The machine could be there for 100 years, it'll never be their doctor in the same way. 

Mark Mullen: And another thing that you talk about is that humans face consequences for our actions. Again, this gets down to the fact that we can feel, and so we can face consequences. And if a human doctor makes an error, obviously there are legal mechanisms for holding them accountable, which you brushed on, but there's also shame, guilt, feelings associated with that, and you just can't expect that [00:17:00] same level of skin in the game, pun intended, from, artificial intelligence.

Jacob M. Appel, MD: Yeah, I, think of that, that scene in, one of the early Star Wars movies where Princess Leia threatens to send R2-D2 and C-3PO back to some kind of drone salvage facility. I'm sure I've gotten the exact details wrong, but the gist of it is all you can do to the machine is turn it off. And we don't even know whether machines have a vested interest in staying on.

When I go home, whether I've made a mistake the patient knows about or simply don't think I've done a good job, I feel really bad about myself. think of it all the times you've gotten calls from colleagues about cases where ostensibly, I actually did things just right, but they're still deeply troubled by it.

And if you're deeply troubled by it as a person, you have the emotional wherewithal often to try to undo what you've done. The machine may decide, "I've done what I've done. Patient is gone. I'll move on 

Mark Mullen: Do you think there are any actions that policymakers need to make to increase the trustworthiness of replicants if they're to replace human physicians in some contexts?

Jacob M. Appel, MD: Absolutely. I think the first thing they need to do is we need much clearer rules [00:18:00] on exactly who is liable for which aspects of machine interactions versus human interactions so we can address, for example, the negative outcome pel- penalty pa- paradox. A- and many of our rules of learned intermediary doctrine simply weren't designed for the current system we have.

and we probably need some kind of ethics oversight board in terms of experts in how we program these machines. It's easier to explain with artificial car-- with, artificial intelligence in cars. So if you're gonna make self-driving cars that are AI-run, the car needs to decide whether it's gonna prioritize the life of the driver in its vehicle or the life of the driver in the oncoming vehicle, and to what percentage.

If you have a one percent chance of your driver dying and a ninety-nine percent chance of the other driver dying, do you still veer or not veer? Otherwise, we have an endless game of chicken. Those same issues in much more complicated ways come up in medicine in terms of triage in between patients and liability.

We need someone thinking through all those possibilities and algorithms, and machines can't do that on their own. 

Mark Mullen: Like an [00:19:00] ethical standard for the AI. 

Jacob M. Appel, MD: Exactly. 

Mark Mullen: One of your assertions in the paper is that the patient-physician relationship is fundamentally an irrational one. I had never heard this relationship described in this way before.

Could you unpack this for me a little bit? 

Jacob M. Appel, MD: Sure. So I, think Atul Gawande, wrote a great column in The New York Times a number of years ago that he was dying of prostate cancer called Doctor Talk To Me. It's a controversial column, but one of the things he really liked is he talked about how he really values looking at the patient's family-- the doctor's family and the pictures on his or her desk, and that makes him seem like a, a family person who he can trust with his disease.

Now, there's no rational reason why having a family on a beach vacation makes you a better doctor, and yet I think that's-- those irrational things that make us connect with people are what make us trust doctors 

Mark Mullen: Leads me to my next question. Other than the provision of medical care, and you mentioned empathy as a part and parcel of this, right?

Like if you're not able to empathize, if you're not able to be authentic, you're going to provide less effective medical care [00:20:00] based on those sort of common factors of healing. But even beyond this, what are some other areas of professional identity for physicians that you think replicants will fail to replicate?

Jacob M. Appel, MD: Sure. So, so the replicant is trained to follow fairly strict boundaries. And yet part of the nature of being a good doctor is knowing when boundary crossings are just crossings and when they are transgressions lead to violations. And there are some patients who I would never think of, walking through the door of a hospital or delivering something at their home, but there are occasionally patients where that would be perfectly appropriate.

Where you have a ninety-five-year-old homebound patient and you want to do them a favor and drop off their medication, there's no concern they're gonna think that you're gonna move in with them or try to take advantage of them. Machines can't make that distinction. So being able to assess the situation in a way that allows you to act on your empathy is really important.

knowing simply not to get the collateral, but maybe there are certain things you would or wouldn't share with the collateral. There are certain things that you would ask that are not relevant to the medical exam, but might reassure the person you're [00:21:00] talking to about their sick relative. 

Mark Mullen: I think also physicians are more than just people who heal individuals, right?

There's advocacy parts of being a physician in terms of professional identity, teaching the next generations of physicians, academic work, which you do a lot of. So I think that while replicants might serve a very important place at the bedside, there are other aspects to being a physician that extend well beyond the bedside that I'm not sure replicants will be able to take part in.

Jacob M. Appel, MD: Yeah. In fact, we haven't even talked about the teaching aspect, but AI may actually make people think less and therefore make them able to teach less and learn less. And there's already a lot of data on that, that when we have AI doing diagnostic techniques, for example, in radiology, the radiologists who start using the technique, particularly the trainees, have lower scores when they try to then do it on their own, which is a huge problem.

Mark Mullen: I've already seen it, to be honest with you. I think it's pretty terrifying. So I think that there is gonna need-- probably there's gonna be a huge backlash, in education I wonder if we're gonna over-correct a little bit before we kind of find that Goldilocks approach to [00:22:00] integrating AI and humans into this.

But definitely I've seen it where taking that mental shortcut of just asking in a large language model to, you know, you dump your thoughts in and you say, "Can you make this into a tight assessment for me?" And I think that has real long-term consequences for 

Jacob M. Appel, MD: sure. Yeah. I don't wanna sound very old, but I think the current generation of trainees has been taught that you don't need to know that much because you can always look it up.

But you need to know what questions you need to ask in order to figure out what to look up or you're in a lot of trouble. 

Mark Mullen: Yeah. And then back to your initial point about how the replicant is not going to have-- It sounds a little bit like common sense, but really it's deep wisdom. It's lived experience of having those soft skills and being able to understand on a personal level what the patient is going through.

I was in an ethics committee meeting just yesterday about a patient who was unrepresented, and the question was: What should we do for this patient? We can't contact their family. They don't have any family for us to contact. They have to make this really difficult medical decision about prolonging the patient's life and weighing that against quality of life.

And I, I-- [00:23:00] my thought in my head was, listening to the ICU doc speak What would you do if it was your sister? What would you do if it was your sibling, right? Like we're talking through all of these, but none of us w- are truly unbiased. And so ultimately, if you're the patient's physician and you have their best interest at heart, you have that lived experience, and you can dive into that to further the dignity of the patient.

And I don't think artificial intelligence is ever gonna come close to that level of deep reasoning. No. But maybe I'm wrong. 

Jacob M. Appel, MD: I mean, we're, it won't be able to bring the, subjective elements of it in a way that is useful. I always like to say it's the difference between having a, pet cat and a pet rock.

A pet rock is a lot simpler, but most people, if you give them the choice, would choose the cat. 

Mark Mullen: I would probably take the rock, but I'm a dog guy. 

Jacob M. Appel, MD: Yeah. 

Mark Mullen: Pulling a lens back a little bit, what do you think is at stake societally if replicants begin to replace human physicians at the bedside? 

Jacob M. Appel, MD: So I think it's, there are two large issues.

One is this is an enormous equity issue because replicants are not gonna replace all physicians. They are gonna be another mechanism in which [00:24:00] people with less opportunity in life who are already limited in social capital are forced, into a second-rate system, and this will be the second-rate system.

People who can buy out of that system will buy out of it. So that's one major problem. but I think the other one, which is another aspect of sort of the dehumanization of our society. I remember when I started at our hospital, it was everything was on a personal basis. We were a small community hospital.

You would walk to the pharmacy and talk to the pharmacist. And step by step, there are computer interfaces, there are go-betweens, there are, are middle managements, and now this is just one more element that takes away the face-to-face element that actually not only helps get things done, but helps get them done in a meaningful way.

Things get done better when people talk to each other. 

Mark Mullen: What have we left unsaid in terms of why it is important for us to see access to a human physician as a right? 

Jacob M. Appel, MD: I, think the most important thing is, there's a lot of talk about loneliness and social isolation and the former surgeon general wrote a lot about this.

People forget that for many people, [00:25:00] particularly older people, isolated people, their doctor is actually one of their most important social contacts. So true. It's a person they trust. Often they actually try to get that person to be their healthcare proxy, which they can't in some states or, even the beneficiary of their will.

That human interaction, which may not seem that important to the doctor who sees hundreds of people, is a lifeline to people, and machines can't do that. 

Mark Mullen: God, you reminded me about one of my favorite parts of being a physician, and es- I'd say especially a psychiatrist. I haven't thought about this in embarrassingly long, but we really do get to interact with individuals who have no other social interaction because everyone has to go to the doctor.

Thank you for that. I'm very moved by that. 

Jacob M. Appel, MD: Appreciate it. 

Mark Mullen: What types of policy interventions do you think are gonna need to happen in order for us to actually ensure this right that humans have this right? What are your suggestions for policymakers if the opinion of Jacob M. Appel is Patients should have the right to a human healthcare provider.

[00:26:00] What's the logical next step in terms of public policy? 

Jacob M. Appel, MD: Well, I mean, I think first off point out that unfortunately in our country people don't even have a right to a provider, period. So that, that is an issue. But assuming we, we assume that people have a right to actually get medical care, which I do believe in, I think the most major step would be if Medicare and Medicaid simply by policy did not pay for interactions that were not primarily human driven, and that any AI augmentation had to be specifically approved, that would be a game changer because that would set a ceiling that insurance couldn't catch up with.

Mark Mullen: Assuming Congress and public policy continues to be mostly useless, do you think that there's any hope that institutions will get this right? That institutions might say, "We're not going to have AI replace physicians. The patient must see a human healthcare provider"? Or do you think that with the invisible hand, any institution that tries to go down that road is just gonna get crushed by capitalism, and so we're gonna have to rely on public policy?

Jacob M. Appel, MD: No, I actually think there is gonna... I think public policy is a better [00:27:00] approach. I think there will be boutique institutions that advertise that way, that say, "Come here and you get to see a human being." Unfortunately, I think there's gonna be a bifurcated market. 

Mark Mullen: Understood and agreed. How hopeful are you on a practical level that we humans are gonna get this right?

Jacob M. Appel, MD: That's I, Without my crystal ball, I can only look into past precedent, which is that we often don't get things right. I, I am-- my hope is not that we get it right at this point, but just not that we get it so poorly wrong that too many people suffer. We'll probably get it somewhere in the middle is my guess.

Mark Mullen: I think the said a lot about that question and about your feelings. 

Jacob M. Appel, MD: Yeah. 

Mark Mullen: Yes. we'll take a quick break, and when we come back, we're gonna dive into how artificial intelligence and healthcare relates to decisional capacity and robo-agents serving as surrogate decision makers.

One of the things that surprised me about being a mental health clinician is [00:28:00] how physically demanding the job is because really we're just sitting most of the day, but it's really hard and really exhausting to actively listen to people for about seven hours a day. So when I finish, the last thing that I wanna do is wrestle with a bunch of different electronic systems to try to get notes done and patients scheduled and billing completed, and that's why I wanna tell you about SimplePractice.

SimplePractice is an all-in-one EHR that is HIPAA compliant, HITRUST certified, and built specifically for therapists. It makes your life easier by bringing scheduling, billing, insurance, and client communication all into one place. SimplePractice has automated appointment reminders that can help reduce no-shows and customizable note templates to make documentation less draining.

And if you're just starting out or growing your practice, they also have a credentialing service that helps you get paneled independently. It takes the total headache out of insurance enrollment, giving you full autonomy over your contracts without any middleman platforms taking a cut. If you're ready to try SimplePractice and simplify the business side of your practice, start with a seven-day free trial and then get 50% off your first [00:29:00] four months.

Just go to simplepractice.com to claim the offer. Again, that's simplepractice.com. Very excited to tell you about a new healthcare news podcast that just came out. It's called Medlines. Medlines is a quick weekly wrap-up of everything in the healthcare field, from breakthrough discoveries and research to politics and medical pop culture.

The best way to think of Medlines, it's kind of like NPR's Up First, which I listen to every morning, but it's for all things healthcare. And honestly, I think a podcast for busy healthcare professionals to stay informed without taking a big chunk of time out of our day is long overdue. Medlines is made strictly by healthcare workers for healthcare workers.

Of course, credibility is key for this kind of a thing, which is why it's led by an amazing team, including the great Dr. Glaucomflecken, Will Flanary, and Lady Glaucomflecken, as well as Dr. Matthew Braddock, who was a producer of Psychiatry Boot Camp. And I'm really excited to hear his writing for Medlines.

The rest of the Human Content Podcast team, who is responsible for Psychiatry Boot Camp, is also responsible for [00:30:00] Medlines, so you can bet that it's gonna be an amazing podcast and hopefully a staple of your feed. Plus, New England Journal of Medicine is on board as the inaugural Medlines sponsor. New episodes of Medlines are out now wherever you get your podcasts, with new episodes out every Friday.

Welcome back. Now we're gonna hone in on a more specific use case for artificial intelligence, and this time we're going to discuss AI, quote, roboagents, which might be used as surrogate decision-makers for a patient. And this question strikes me as acute. as you write in your paper, we already have dynamic AI models that patients are relying on for an endless array of functions, and it's not a stretch for me to imagine that someone might appoint an artificial intelligence model to be their surrogate decision maker in the case of decisional incapacity.

In fact, I'm actually more surprised that I can't find a news report about this already happening. 

Jacob M. Appel, MD: My guess is it's probably [00:31:00] already happened and it just hasn't hit the news yet and it's gonna surprise one of us any day now. 

Mark Mullen: Yeah, I think so too. And I wanna remind our listeners that we're talking strictly about patients who have been found to lack decisional capacity for a certain decision, in which case we as healthcare professionals need to decide what to do for the patient by other means.

So could you please briefly run us through the existing avenues for ensuring patient autonomy in the case of incapacity? 

Jacob M. Appel, MD: Sure. So once the patient's determined to be-- to lack capacity, we then look for an advance directive, which either could be a healthcare proxy designated to make decisions or a living will, which is an out-- a detailed description of what they want done under certain circumstances, or a combination of both.

And obviously in the absence of either one, then we would turn to a surrogate or in some states it's called a next of kin to make decisions for them. What I wanna point out is what I am not talking about are people who use AI or chatbots to design their living will and [00:32:00] then bring it on paper to the hospital or on their phone to the hospital.

I'm talking about people who rather than appointing a proxy or healthcare agent to make decisions for them, want the chatbot itself to make the decisions. So the chatbot will show up, they will be incapacitated, they will have shared with the chatbot what they want to some degree about their life or their history, and the doctor will plug information into the chatbot and come up with an answer that mirrors the patient's theoretical preferences.

Mark Mullen: And this is what you're calling a robo-agent would be this chatbot and we're gonna talk about kinda what exactly this might look like in terms of training a chatbot and asking it to, do your will for you. Let's then talk about what you mean when you say that you might be able to use artificial intelligence to enhance your advance directive short of a robo-agent.

So what would that look like? 

Jacob M. Appel, MD: Sure. So, so I, will emphasize by the way, because I know the article I've written says robo-agent. I love the term robo-proxy, but the reviewers were concerned that in some jurisdictions there are no proxies, hence the term, but I'm hoping the term [00:33:00] robo-proxy catches on because it has a better ring to it.

Mark Mullen: We'll use robo-proxy here. 

Jacob M. Appel, MD: I appreciate that. So I think right now people are using AI all the time to sit down in front of their computer and asking it something like, "I need to write a living will. what factors should I include?" And the machine may ask you, "Well, what do you care about?" And then it helps you to write one.

so and I've seen those. I've actually seen people with printouts from their machine. And those are perfectly legal in every state. There's no reason to doubt them. But this is a very different mechanism because this we can't ask a patient whether or not they agree with the statements. It's algorithmic and I can go through but there are a lot of structural problems with it.

Mark Mullen: Do you think that artificial intelligence tools actually do offer a benefit over existing tools For creating that living will? Like, do you think there's something unique about artificial intelligence that is going to help people to make more specific living wills that further their wishes aside from what they might do right now, which is probably filling out a template, maybe talking to a clinical ethicist or a lawyer?

Jacob M. Appel, MD: So I think in an ideal world where people are talking to a clinical ethicist and a lawyer and actually thinking through these issues, [00:34:00] that is probably preferable to what AI is gonna do through a chatbot. That being said, I think most people don't do any of that. They are told to print out a form and fill it out, and they check it very quickly without thinking about it very much.

Mark Mullen: Right. Do you wanna be intubated? Do you want CPR? Do you want trach and PEG? Fair enough. I guess what I'm-- was sort of trying to lead you to is, do large language models have a better ability to page through someone's medical record and pose very specific questions about, these are your-- like, these are the specific medications that you're on.

These are the specific preexisting conditions that you have. These are your specific risk factors and how things are likely to look, and so do you think they can get more granular than a clinical ethicist and a lawyer, or do you think maybe they could get more granular, but that might not be an advantage?

Jacob M. Appel, MD: So I think they probably could get more granular. I think rea-realistically, most people are not using them when they write their living wills in that kind of way where they're programming their health record. They're just asking it for advice and programming a few random facts, so it may be less productive because they're cherry-picking the facts.

But I'm not sure how productive it is because a human ethicist is very good at [00:35:00] figuring out which things really matter and which ones don't, and the machine may view all factors as equal 

Mark Mullen: Why would a patient want a robo-proxy to be their healthcare proxy rather than a human proxy? 

Jacob M. Appel, MD: Well, the most obvious reason is you might not have a human being to rely on.

We used to use the term "unbefriended patient." I'm sure there's a more socially appropriate term for it now. But people who are socially isolated, older, have outlived their spouses, their families, may not have someone else they can trust. Right now, occasionally, those people come to me or come to their doctor because I'm the one person they know.

The chatbot is an alternative, and they can spend a lot of time with the chatbot because the chatbot has an infinite amount of time to spend with them to gather information about their lives if they want to. 

Mark Mullen: So a chatbot has unlimited time. Chatbot is available twenty-four/seven to act as a proxy as opposed to a human.

it works for-- I, use the phrase unrep- we, I should say my institution uses the phrase unrepresented patients to describe these patients. What about other concerns that someone might have turning over such power to a human versus a [00:36:00] robot? 

Jacob M. Appel, MD: So I think one of the major concerns in turning things over to a human is you have to trust them.

And I can tell you that-- and actually, not only do you have to trust them to act in your best interest or do what you wish them to do, you have to trust them to get it right. And there's a lot of data out there to show that, even in situations where people have discussed their healthcare wishes with their third-party decision maker, the third-party decision maker a few months later either no longer remembers it or doesn't fulfill it the way they want to.

So human beings are not that reliable in this regard. A-a-and I will add human beings also have their own motives and may wish to keep grandma alive longer than Grandma wished to be kept alive in certain circumstances, which is human nature. But on the other hand, like you mentioned, the major downside of turning things over to a robot is the robot uses an algorithm, and ninety-nine percent of the time the algorithm may get it right.

But one percent of the time the algorithm may give an answer that is wildly wrong. We have no way of knowing which of those situations it is, and even when it seems wildly wrong to the doctors involved, we [00:37:00] can't ask the patient because they lack capacity, and we can't ask the robot how it got there.

Mark Mullen: Yeah, I think that those examples at this point have stacked up. They're countless, and everybody, I hope, is aware of them, where AI models are fa-- like if, they're writing a legal brief, they are fabricating the case law that it's based upon. If they're writing an academic paper, they're making up articles to support their claims.

And so I think it's the hallucination's a bastardized term from us psychiatrists. I think they continue to be fairly rampant. And even if they're minimal when you're dealing with the decision to be able to make decis-- you know, when you're dealing with high stakes like this, life or death stakes, we really can't afford that one percent, glorious error.

Jacob M. Appel, MD: They're worse when they're minimal in a way. Because when the algorithm says the moon is made of Swiss cheese, I know that's not true. When the algorithm says that two point six percent of the moon is bauxite rather than two point eight, I'm just making it up, I have no idea whether that's true. When the algorithm says, "If I'm on a ventilator, I want you to set me on fire," I can assume that's not true.

When it says, "I only wanna live in this condition and not that [00:38:00] condition," it seems plausible. How would I know it simply made it up? 

Mark Mullen: Yeah, you have no way of knowing. 

Jacob M. Appel, MD: No way of knowing. 

Mark Mullen: Mm-hmm. So you list four different ways that patients might choose to use robo proxies as their surrogate decision makers.

The first is traditional living wills with chatbot applications. What might that look like? 

Jacob M. Appel, MD: I'm trying to remember exactly what I meant by that specific terminology, but you could basically design a living will the traditional way through a chatbot. So that's the most basic level. The second level of intervention is where you have general parameters of what you want done, but in those parameters, the chatbot can sort of fill in the blanks in the same way if you had both a proxy and a living will, the proxy could fill in the blanks where the healthcare document doesn't do, doesn't like - 

Mark Mullen: First one is extensively trained.

You've personalized the model. You've spent time with the model training it. The second one is you submit a document and you say, "Follow these rules." 

Jacob M. Appel, MD: Yes. 

Mark Mullen: What's the third one? 

Jacob M. Appel, MD: So the third one is, the possibility that we simply have chatbots out there that are marketed as healthcare decision-making algorithms [00:39:00] that get a certain amount of data and aggregate a certain amount of sense of what people want in the community on average.

And they could be used, for example, for people who've never had decision-making capacity that we know of or we have no idea what their wishes are, to sort of aggregate the reasonable person standard and apply data as it comes to you. 

Mark Mullen: And even in this case, I think you're giving us modern humans more credit than we deserve because I think it's more likely that someone will just say, "Hell, I use ChatGPT forever."

I mean, ChatGPT isn't designed as a therapist, but a lot of people use it that way. I think people would say, "ChatGPT is my proxy," and it won't even be something that is specifically trained to be a healthcare proxy. Do you think that's possible? 

Jacob M. Appel, MD: I think that's very possible. And I, think I add to that if you just say I wanna have, AI as my algorithm and we don't know which chatbot, that could be more complicated because, I'm, no expert on different kinds of chatbots, but different ones will come up with different answers to the exact same scenario.

Mark Mullen: Yeah. It's almost such a vague living will as to be useless. I would hope that a judge would even just thr-throw it out and say, "This is not specific enough to even be acted upon." 

Jacob M. Appel, MD: I mean, you [00:40:00] could... I, always like to say you can imagine two sisters coming in for their brother and one saying, "He wanted us to use Anthropic," and the other one says, "He wanted us to use ChatGPT or OpenAI.

How do we know which is right?" 

Mark Mullen: Mm-hmm. Yeah. As a clinical ethicist, when you do get into this situation where, let's say, you have two surrogates of sort of equal rank that are not agreeing, putting on your clinical hat, how do you navigate those conversations? 

Jacob M. Appel, MD: So clinically, I think the-- in most states and ethically, the goal is to figure out what the patient really wanted.

And usually sitting in a room talking to people, you can piece together some kind of consensus about what the patient's goals, values are versus what the stakes of the other players are. And often it'll become pretty clear that everybody agrees grandma wanted one thing, but for different reasons, other parties may not want something else.

But the goal is also to these individuals, both whoever, they are, have to survive with the outcome long after the patient is no longer around. They have to live with the result. So trying to find a result that both meets the patient's autonomy, but also is livable with for the survivors. [00:41:00] 

Mark Mullen: So are you saying that you as the ethicist are trying to sort of uncover the common nugget of truth in what both of these metaphorical sisters are saying But then ultimately you may have to force one to just kind of go with what you think is the truth?

Jacob M. Appel, MD: Well, I don't force anybody. I always say like courts force people. My goal is to help people figure out what they really want. A, to build consensus, but also even beyond understanding common ground, to be able to reshape or reframe things if somebody may not get the outcome they want, but they still feel comfortable with the outcome.

The machines can't engage with an epistemic that way because they can't pick up the kind of nuance, they can't build the consensus, and they can't figure out how to phrase things in a way that people may not get the outcome they want but still feel like they're okay with the result. 

Mark Mullen: So the fourth thing that you mentioned is someone might appoint a robo proxy and also a human proxy and say they need to share it, they need to build consensus.

And now what you're telling me is that you run into this problem fairly frequently, I'm sure, two human proxies [00:42:00] that disagree, but you don't even... You think that the skill set that you've developed to navigate that conversation will not translate well into navigating that conversation between a robo proxy and a human.

Jacob M. Appel, MD: I think it will be impossible. I don't think you can negotiate with a chatbot. I mean, the chatbot may be programmed to bend over to agree with you or trained to be stubborn, but it's not really meaningfully engaging with the conversation. And so you're really having a one-way conversation with a patient, and the chatbot is sort of like the pet rock.

Mark Mullen: You had mentioned, we had mentioned that one reason that a human might turn to a chatbot for this is that they don't trust a human proxy to be unbiased and set themselves aside and, put themself in the patient's shoes. And, you know, 'cause humans, especially people that know someone well enough to be a proxy, do have these kind of messy ulterior motives, whereas an AI may not.

So I'm talking-- I'm trying to talk about bias here a little bit. Do you think that these large language models actually are more unbiased than the human counterparts? 

Jacob M. Appel, MD: So the [00:43:00] best answer I can give you is we don't know because we have no way of assessing how biased the AI models are or no meaningful way in any particular situation.

I imagine they do have biases built into them because on the one hand they're programmed or trained by humans, but they're also aggregating mass amounts of data that have inherent biases built into them. So if we live in a world with structural racism and inequality of capital, those structures will all be built into the data the AI is mining, and therefore the AI will be biased.

We just don't know how it plays out in a given setting. 

Mark Mullen: Can't compare on a one-to-one scale, it's kind of apples and oranges is what you're saying, but certainly AI is not unbiased. 

Jacob M. Appel, MD: No. We-- The difference is with human beings we can often with some work figure out what the biases are, and with AI we can't even figure out what the biases are.

So it's in like a lot of the recent work in DEI, one of the goals has been to make people aware of their biases. You can't make the machine aware of its bias. 

Mark Mullen: So in terms of benefits to using a robo proxy, we talked about the unrepresented patient. We talked about hoping that [00:44:00] the proxy is not going to bring its own personal biases to the table, but understanding that in fact it might.

Any other major benefits to using a robo proxy that you want to make sure we mention? 

Jacob M. Appel, MD: Yeah, I mean, I think the most basic one is robot proxies are much easier to use than human beings, and that's why I think they're gonna catch on. 

Mark Mullen: You do think they're gonna catch on. This is not some academic- No

article for fun. You think this, this is hap- gonna happen on a large scale. 

Jacob M. Appel, MD: I think it's gonna be a real issue. I mean, some things I write about, I think they're sort of theoretical. I think this is actually coming down the pipe. I think a lot of people, particularly the younger generation, are gonna be using these.

In fact, I think one of the unintended upsides of this, maybe for the first time, younger people and a generation that generally has been tuned out of doing any advanced care planning may feel, "Well, it's really easy to do with a chatbot. I use my chatbot all the time. I'll use it for that anyway." So it may actually bring more people into the designing advanced directives field.

That being said, I don't think that outweighs the, serious concerns. 

Mark Mullen: Sure. So let's go into those then. I'll just pitch you [00:45:00] them kind of as you, you lay them out in your paper, so I'll just pitch you one by one to ask you to go into it. So let's start with informed consent. 

Jacob M. Appel, MD: Sure. So most people have no understanding of how chatbots work, including the people who design them.

So how you can inform someone of the risks of being involved in this, the, the possible downsides i- it's simply not doable. Not to mention which nobody's actually sitting down with a person when they agree to use the chatbot and doing any form of informed consent process like you would with any other medical procedure or device.

Mark Mullen: I love this next one. Replicability, reliability? 

Jacob M. Appel, MD: Yeah. So if you program the exact same information into a chatbot 100 times, you will get probably 97 to 98 very similar answers and one or two very different answers, and that's an enormous problem because you don't know... Not to mention which, when we use the chatbot, since we might get different answers, do we use it once?

Do we check again to see whether it comes up with the same answer as a doctor? Do we run it through 100 simulations and take the majority one? most people are not gonna [00:46:00] specify how many rolls of a dice, so to speak, they want with their chatbot. 

Mark Mullen: Yeah, and I think you could argue that humans might also be inconsistent decision-makers, but the process by which humans arrive at those decisions is so fundamentally different and non-comparable to the probabilistic nature of a large language model that is just by definition going to vary responses a certain percentage of time.

Jacob M. Appel, MD: A- and if somebody says to you when they, want one thing done for their grandmother or things she always wanted, and 10 minutes later they say something different, first you can actually ask them, "How did your opinion change?" and they credibly make explaining it, and then you may be able to disqualify them as a proxy if they tell you a different thing every five seconds.

The machine is by definition gonna do that. So every machine would need to be disqualified if you use that logic. 

Mark Mullen: Wow. What about impact on family of people who use robo-proxies as opposed to human proxies? 

Jacob M. Appel, MD: So I think this could be devastating, and it's really an underappreciated value. Somebody- I'd be 

Mark Mullen: pissed.

Jacob M. Appel, MD: Yeah. Somebody appoints th- their chatbot as their proxy and the chatbot says, "Take me off the [00:47:00] ventilator," and your spouse says, "I'm married to him for 30 years. That's not what he wanted." But the chatbot's the decision-maker That's the problem 

Mark Mullen: Yeah, it's pretty gut-wrenching, I think. it's offensive, I think, actually, to...

I mean, I don't know, it gets at the nature of love, right? And what we rely on our loved ones for. 

Jacob M. Appel, MD: I'd like to think that no matter what you wrote in your, advance directive or your instructions about using a chatbot, in that actual situation, the judge would look at the case and still overrule it, but you never know.

Mark Mullen: Do you think that's most likely, just based on your legal training? 

Jacob M. Appel, MD: I think at least in some sense it, some cases it will. I think no matter what a machine says, when a, a spouse is coming and pleading with a judge, judges are human beings, at least for now. Obviously, if we end up with chatbot judges too, that will be a different world.

Mark Mullen: Yeah, I was, raised by a judge, actually, so we, I'll ask him over dinner tonight and see what he would say about that. What about impact on doctors? 

Jacob M. Appel, MD: I think it's gonna be demoralizing for doctors. First, it's gonna create lots of ethical challenges for doctors, but it also is gonna create a sense of demoralization, is basically the way I would describe it, because you may think that something is [00:48:00] clearly the right answer, and with a human being, you can negotiate that.

With a proxy, you can't negotiate with a machine. 

Mark Mullen: Yeah, and you had mentioned in our last, I think, artfully in our last conversation about how sometimes in limited circumstances when there clearly is a decision that needs to be made clearly in the best interest of the patient, the physician can, and at, to a proportional degree to how clear it is that this is in the patient's best interest, not coerce, but persuade a surrogate decision-maker that this is really the right thing, provide some education, provide some motivation, versus any of those tools are just simply not gonna work on a, robo-proxy.

Jacob M. Appel, MD: Yeah, I mean, I, and I don't do this often, but there are times when I'll say like the two options here are letting nature take its course, which I think would be a very reasonable thing to do, or if you want to engage in heroic measures, they candidly could be extremely painful and invasive. Now, that's a very skewed description of something, but if I say that to a person, they will allow me to let nature take its course 99% of the time.

A [00:49:00] machine is not gonna be influenced by that. 

Mark Mullen: And then I'm gonna drill down on this one one more time because I was very interested when I read this portion of your paper. So, and this is a genuine question just as someone who works in a CL setting, sits on the ethics committee, 'cause this comes up not infrequently.

So if robo-proxies are given rigid guidelines by the patient Chances are that we're gonna see increasing instances where the physician does not necessarily agree with the RoboProxy based on the reasons that you just said. One of the things that you write is, "The possibility exists that physicians will be compelled to implement medical decisions which they strongly disagree with and then defend these decisions to family members who also object to them," because the, the RoboProxy is sort of ruling all of us.

Now, my understanding is that if in the opinion of a physician the risks of an intervention outweigh the benefits, the physician is under no obligation to offer that intervention. What am I missing? 

Jacob M. Appel, MD: So that's g- it's often true, but not always true. So for example, if the patient says, "I want you to turn off my life support in [00:50:00] a certain circumstance," even if you strongly disagree with that decision, if they have capacity, you'll turn off their life support.

If they, say in an advanced directive very clearly, "I don't wanna live in certain circumstances," even if you think a reasonable person would, you'll still turn off care. If they have capacity when they make it, and they say, "If, I reach the age of 50 and I have, I don't know, pneumonia, I don't want antibiotics," we'll honor that.

The problem with a proxy, with the, with a robot, is the robot is gonna say things just like that all the time that seem very unreasonable to doctors, and we're gonna honor that without being able to disagree with them or negotiate it, even when the doctor and the family are both on the other side. 

Mark Mullen: Is the reverse-- Again, this is just authentic from me.

Is the reverse also true? So in the case of someone who, as a physician, you would now say, the benefits to ongoing life support are relatively minimal. This person is in pain, and we're just prolonging their life. The risks are significant that they're going to be in more pain, and I no longer think that the benefits of whatever it is, life support, trach and PEG, outweigh the risks.

Does the physician [00:51:00] in that case have the right, to just say, "I'm no longer offering this intervention, and I'm gonna withdraw life support"? 

Jacob M. Appel, MD: So it's interesting you mention that because it's, not a perfect balance. If patients don't want more care with very few limits, we can't give them care they don't want.

But if patients want excess care, at some point we can say this is medically futile or well beyond the standard of care. On the other hand, there are gonna be a lot of areas where patient may be demanding something that is not truly futile, but seems like a very poor choice, and we still honor that. The proxy may make the same decision.

The proxy, for example, a patient... If a patient said, "I'm willing to try this very unlikely cure even though I know it will cause me horrific pain," we would honor that. If the robot says that, we would be obliged to do the same thing even though the patient is howling in agony and the family and the doctor may not think that's really what we want.

Mark Mullen: We may honor that, but we may also say, "Well, I understand that you wanna take this 1% chance and have me do this surgery, but I'm not gonna offer that because I don't [00:52:00] agree that the benefits outweigh the risks," right? 

Jacob M. Appel, MD: I mean, I'm-- There will be some tipping point where we don't offer it, but you can make it 5% or 10%.

There is some area where the patient's wish to have it done will be honored even though it may strike a doctor as not something that a reasonable person would want. 

Mark Mullen: I feel like this is the conversation that we're often having in the ethics committee, so I, appreciate the opportunity to have that conversation with you.

And then going further into the drawbacks of using a robo-proxy, I saved the best for last in my opinion. You talk about the decisional standard. So I think first you're gonna need to teach us about what it means to have a decisional standard and the substituted judgment standard versus the best interest standard.

Jacob M. Appel, MD: Sure. So in any decision, courts always ask what threshold of certainty we need to have. Do we need to have a preponderance of the evidence, meaning it's more likely than not this is true? In criminal cases, it's beyond a reasonable doubt. in most medical cases, in New York State at least, it's clear and convincing evidence, which is somewhere between, it's like a rough 80% maybe, some people [00:53:00] say 85%, somewhere in that ballpark.

So how certain do we have to be that the AI, when it expresses a wish about you, crosses that threshold? If the AI is right 93% of the time, is that gonna meet the clear and convincing threshold, clear and convincing standard if it's right 86% of the time? That's gonna be an enormous problem for people to figure out because it's gonna be hard to pinpoint this with very much certainty.

Mark Mullen: Can you tie that into the substituted judgment versus best interest standard for me? 

Jacob M. Appel, MD: Sure, absolutely. So the, the other question that courts have to generally address is whether or not in patients who lack capacity, we are using a substituted judgment standard of what they would want done or a best interest standard, what society as a whole would want in that situation or a reasonable person would want.

And most states have taken a substituted judgment standard. The problem is for patients who've never had capacity, patients, for example, who are children, patients who are cognitively impaired, and there's a significant divide. Some states, like Massachusetts, actually use a, substituted judgment standard [00:54:00] anyway.

They ask people to imagine if you were in this pers- if this patient were in their situation and were cognitively impaired, what would they want done? Other states like New York view that as a bridge too far. AI is going to open a field day and actually a m- a series of minds in this particular area trying to figure out substituting itself for people who've never had capacity what they would want done.

It will... The, possibilities of getting this wrong are endless. 

Mark Mullen: It's interesting that you bring up the, how this is actually a, something that's ruled by the courts and, legislatures because I, wasn't thinking of substituted judgment versus best interest in that way. I was thinking of these as not always competing, but, sort of different ways that a surrogate might approach making a decision for somebody else Are you-- Am I understanding correctly that in most states it's actually mandated that a surrogate decision-maker use a certain one of these standards when making decisions for a loved one?

Jacob M. Appel, MD: Yes. 

Mark Mullen: [00:55:00] Wow. 

Jacob M. Appel, MD: So, so, so New York, State at least, but it's most states, but in New York State, it's very clear that the proxy or surrogate is supposed to use the wishes the patient would have if they had capacity in that given circumstance. 

Mark Mullen: I didn't know that. I thought that, there was more flexibility than that.

So it's actually mandated by law that-- I knew it was like the consensus of ethics organizations and palliative care organizations, but I didn't realize that it's actually legally mandated to use that standard. And then it-- and, I guess there are then-- your point is there are cases where there is no substituted judgment standard because there sort of was no, informed consent judgment to begin with, and so in that case you have to use the best interest standard.

Jacob M. Appel, MD: Yes. a-and in that case, I think it's gonna be, with some states like New York use the best interest standard. Others still use substituted judgment standard. The case is, I think it's Psychowics is the Massachusetts case versus Belchertown, and that involves a lot of speculation and some people think a bridge too far, but that's gonna be a ripe area for AI to be involved, putting itself in the shoes of [00:56:00] incapacitated patients to think what they would want if they'd had capacity.

It's-- I, I know it's very meta, but AI is gonna thrive on that, for better or for worse. 

Mark Mullen: Yeah, for sure. And then just for my own education, could one-- could a family member then sue a surrogate decision-maker and say, "You were supposed to use the best interest standard, but you clearly used the substituted judgment standard, and therefore there are damages"?

Jacob M. Appel, MD: Very unlikely. 

Mark Mullen: Yeah. So th-this is a little bit more abstract, right? Yeah. Like a little more theoretical. 

Jacob M. Appel, MD: I mean, most states proxies and surrogates have a complete safe harbor as long as they acted in good faith. 

Mark Mullen: Okay. Gotcha. So it is, legislated, but in terms of the enforcement of the legislation, there's a safe harbor.

Jacob M. Appel, MD: Yeah. 

Mark Mullen: Okay. Thank you. Okay, now we're gonna keep your lawyer hat on and talk about liability. Why should physicians care about the question of liability when it comes to robo-proxies? 

Jacob M. Appel, MD: Because they're gonna get sued is the short answer. Because the reality is you could imagine that situation where the robo-proxy says, "Take my-- take me off [00:57:00] life support," and the spouse says, "He's young, he's healthy, he's fifty, he's gonna recover.

He would never want that." And the hospital or the doctor follows the machine. You can be confident whether or not they win that lawsuit or lo- lose that lawsuit, they are going to be sued. A-and I would not wanna be on, the doctor's side of that case because no matter what the law says, when you explain that situation to a jury and there's a person's young spouse crying in the jury box i-in the witness stand, it's gonna be a bad case to defend.

Mark Mullen: So what are your recommendations in terms of improving these liability statutes so that we are ready as a healthcare system when robo-proxies are used? 

Jacob M. Appel, MD: Sure. So I, think- On the one, on the front end, you may wanna in-build in increased safeguards for how robo-proxies can be designed. So maybe people should not be able to make a robo-proxy at home on their own.

Maybe they should have to sit down with their doctor and do it. Maybe they should even sit, have to sit down with the person who would [00:58:00] normally be their surrogate decision-maker in the absence of a robo-proxy if they have one within a certain degree of collateral and discuss this and get them to at least acknowledge it.

So that'll prevent some of the litigation in advance. Obviously, it won't prevent all of it. 

Mark Mullen: Who will invalidate robo, robo-proxies that are appointed outside of these safeguards. 

Jacob M. Appel, MD: Exactly. You might also... I mean, there are specialized judges and specialized juries. This might be a situation where you wanna have these cases, which are, probably not gonna be the kinds of cases that require juries under, under the Constitution or under statute, to actually be heard by judges, preferably, because that will take out at least some of the bias of jurors seeing a, a dead patient and a very reasonable-sounding suffering spouse.

Mark Mullen: Any other public policy interventions that you think we have to, do immediately to prepare for this? 

Jacob M. Appel, MD: I mean, I... Rather than going into the specifics, I think the most broader thought is we have to do something. Right now, there is no legislation in any state to my knowledge being even proposed to deal with this issue, and we really don't wanna deal with it after the fact.

Mark Mullen: Yeah, this is gonna happen, and we're gonna read about it on the news. I have a [00:59:00] feeling that w- And then they're gonna read your article and say, "Gosh, we should've made public policy about this." My last question for you, I think, is probably the most interesting one on this topic as well. So traditional advance directives are effective immediately, and then they apply to decision-making when the patient loses decisional capacity.

And then importantly, these advance directives can also be canceled by the individual who created them at any time as long as, of course, that individual still has the capacity to make those decisions. It would seem to me like a robo-proxy would be the same, but you have some concerns about this. Why are you concerned that this will be different with robo-proxies?

Jacob M. Appel, MD: So, so one concern I have is that people may not wait to activate their robo-proxy until they lack capacity. People may actually bring their robo-proxy with them to the hospital and say, "Doc, don't talk to me. Talk to the robo-proxy." And I'm not sure no matter how much you understand that, you, can't contract outside the standard of care.

I would like to say normally we'd do surgery in the operating room with a scalpel. Even if you have full [01:00:00] understanding we can't do it in the lobby with a cleaver at half price, and this is sort of like doing it in the lobby with a cleaver at half price. We may not be willing to let you drop below that standard, but people are gonna request that.

And there's something very different about someone saying to me, "My spouse is here. I've married to her for 30 years. Talk to her. I'll listen," versus, "The machine's gonna decide. I'll watch." 

Mark Mullen: Yeah, I very much agree. And then tell me about this self-preservation instinct. This got a little dystopian. 

Jacob M. Appel, MD: Yeah, so I, mean, the, the, proxy may also, I mean, the, the robot pro- robo-proxy may have an also interest in continuing its role as robo-proxy.

So for example, it may not wanna turn off life support because when it turns off life support, its significance is no longer there 

Mark Mullen: That's wild. or you might say, you know, let's say that the person who initially enacted the robo-proxy changes their mind and disables the robo-proxy and then becomes incapacitated.

And so you might say to the robo-proxy, "You're no longer the robo-proxy." And they might say, "Well, yes. Sure I am." This person, they might just make it up just 'cause they have that very [01:01:00] interesting self-preservation instinct that we've noted in a lot of other domains, especially like cybersecurity and things like that.

Actually, no, the, the... I am still a proxy. I really am, and, here's why. I think that's pretty fascinating. 

Jacob M. Appel, MD: Yeah. It's kind of terrifying. 

Mark Mullen: This whole conversation has been kind of terrifying. Do you have any final thoughts for our terrified audience? 

Jacob M. Appel, MD: Yes. You should make yourself an advance directive the old-fashioned pen and paper way, and talk to someone you love or care about of what you want done so this doesn't happen to you.

Mark Mullen: Any other final thoughts? 

Jacob M. Appel, MD: I think that's my limited wisdom. We live in a brave new world, and I'm exploring it with all of you. 

Mark Mullen: Thank you, Dr. Jacob M. Appel, and I look forward to having you back in the future hopefully. 

Jacob M. Appel, MD: Thank you

Mark Mullen: Thanks so much for listening to this episode of Psychiatry Boot Camp. If you're enjoying the show, I would love to know what you think. We especially appreciate ratings and reviews on Apple Podcasts or ratings on Spotify. If you wanna connect with us on TikTok or Instagram, we're there @psychbootcamp. You can also email me anytime, [01:02:00] mark@psychiatrybootcamp.com or visit us online, psychiatrybootcamp.com where you can sign up for our new newsletter.

You can also connect with the rest of the fun Human Content Podcast family on Instagram and TikTok @humancontentpods. We also have full video episodes of our show available now on YouTube at Psychiatry Boot Camp. Thanks again for listening. I'm your host, Mark Mullen. Our executive producers are Aron Korney, Rob Goldman, Shahnti Brooke, and me, Mark Mullen.

Season Four is produced by Amana Janjua and this episode was initially outlined by medical student Josh Garrett. Our editor and engineer is Jason Cortizo. Theme music was generously donated by Cave Radio. You can find Cave Radio on Spotify and other music was by Omer Ben-Zvi. To learn about our program disclaimer and ethics policy, our submission verification and licensing terms and our HIPAA release terms, go to psychiatrybootcamp.com or you can reach out to us with any questions or concerns as well.

Psychiatry Boot Camp is a Human Content production.[01:03:00] 

Hey everyone, thanks for watching. If you enjoyed the show, please remember to subscribe to the channel. If you'd like more episodes, you can click right here. I'd love to connect with you more, and I'm looking forward to talking to you next