No Aggregation Without Representation: Patients, AI, and the Fight for Data Rights

[00:00:08] Welcome to the first episode of Joy, Love, and Grace series on HIT Like a Girl Pod. I'm Grace Vinton and I'm honored to welcome to the mic Andrea Downing. She's a security researcher and founder of The Light Collective. Thanks for joining me today, Andrea. Thank you, Grace. I'm so honored to be here. Long time fan, first time caller. Oh my goodness. I am a long, long time fan of yours as well and I cannot wait to hear about your journey. So tell me a little bit about the career path that brought you to your role in the world.

[00:00:38] What are you doing today and ultimately founding The Light Collective? Sure. So I like to explain to people that I have a bug in my code. Are you familiar with the breast, we call it the breast cancer gene, but it's actually a mutation on BRCA1 and 2. So a long time ago, back in the old days of 2005, I tested positive. And this was the early days of genetic testing and really awareness around BRCA and pre-vibers.

[00:01:06] So I found out I had up to an 87% chance of breast cancer and up to a 60% chance of ovarian cancer in my lifetime as a young woman. And a lot of my early path was navigating without the resources, without the support, without the community that eventually started to thrive on social media in 2013, 2014.

[00:01:28] My path was all around being able to apply, I think this combination of lived experience, which a lot of patients come from with my professional background where I started my career in tech.

[00:01:42] I'm kind of a nerd. And I just love to nerd out on things. That was the backdrop for the beginning of digital advocacy and wanting the good type of data sharing and innovation that we want to see not only in the cancer community for better options to screen for, treat and prevent cancer. But then we started seeing some of the downsides of how technology can be used.

[00:02:09] And that is where we started the Light Collective. So for those that don't know about the Light Collective, I want you to tell them a little bit about what your mission is as an organization and what you do. Sure. Yes. The Light Collective is a patient-led organization. Everything that we've done, everything that we've designed is 100% patient-led. And our mission is to advance the rights, interests and voices of patient communities in health technology.

[00:02:37] The way that we say that, the nerdy way in short, is no aggregation without representation. So this idea that in healthcare, often we go on these shiny stages, or when I say we, we have a lot of powerful people going on stages and talking about patients, talking about the data that is really important to be able to make informed decisions.

[00:03:02] But how do we help ensure that patients are not only part of the design and technologies that serve us, but part of the governance? That's why we say no aggregation without representation. Amazing. Now today, you and I are going to talk about cyber security, cyber resilience, a really hot topic in the industry and something that you guys are addressing head on at the Light Collective. But I want to go into a little bit about what's happening in the industry.

[00:03:28] Can you tell us a little bit about Mythos and what happened there? Sure. So Mythos is a new frontier model from Anthropic. And I would say, let's turn the clock back to March and April. There was a big announcement from Anthropic that they were releasing a new model that was way more powerful than any other model that had ever been released.

[00:03:57] And one of the big parts of the news story there that I started catching on to at the time was, I forgot to tell part of my story. Let's just take a step back. And yes, I am a tech nerd. I am a patient advocate.

[00:04:12] But in 2018, I also kind of fell down this rabbit hole of becoming an accidental hacker and connecting with the broader cybersecurity community and learning some of these places that we really need to protect our critical infrastructure, health information, ways that those things can be weaponized. So that's kind of a backdrop from the last seven years. Spoken at DEF CON five times. I've navigated some pretty major coordinated disclosures.

[00:04:42] When you find a bug or a vulnerability, that backdoor in a system, you need to ethically disclose it to the company so they can patch it and protect people, especially in healthcare. So giving that backdrop before I talk about mythos. And in that period from when we started the Light Collective to this last year, we've really been working to build community understanding of what benefits of technology are. I am by no means a Luddite.

[00:05:12] Some people think, oh my God, Andrea's talking about the risk again. And is she anti-tech? No, it's really when you know enough to know how things can be weaponized. You really want to make sure that communities that are adopting new tech can do it safely and in ways that are worthy of our trust. And we've seen a lot of examples where it hasn't been worthy of our trust. So along comes mythos in April, March to April timeline.

[00:05:39] And basically what they found, and they created this thing called Project Glasswing, which is a consortium of companies like Amazon, Google, like all the big tech companies and the cybersecurity people. They brought them all together and they were like, oh my God, this can find the backdoors and vulnerabilities in systems way faster than anything that has ever been seen in the history of cybersecurity.

[00:06:07] And there's this thing called the zero day clock. I think this is really fun. Let me go find this. So zero day clock, here's a link. And people in cybersecurity have been tracking the amount of time that it takes for us to find a vulnerability and then exploit the vulnerability before it gets patched. Right.

[00:06:30] So the timeline is back in the early days when I first started in this space, it was weeks, months, sometimes years for an attacker to be able to find one of these vulnerabilities or backdoors and then use it to either brick a network, tank a system so you can't use it, or scrape all the data and then hold it for ransom. Those are basically the things that bad guys like to do.

[00:06:57] And so what the white hats in this community are trying to do is defend that infrastructure. And the timeline for defenders to be able to patch vulnerabilities has basically collapsed to zero. And that is because of where we are with the power of these models and being able to use them to kind of scan for open misconfigured content management systems.

[00:07:23] You don't have to know a lot about cybersecurity to know that's kind of a big risk in healthcare because we're dealing with people whose lives depend on technology, right? And data. We want to make sure it's safe. Wow. And so what exactly happened with this model?

[00:07:42] With Project Glasswing, these experts in healthcare cybersecurity that I know and have had access to this model have basically said, okay, hackers going to be hacking a lot this year. And we're already seeing some of the examples of that where there was a hospital in Boston that completely got shut down. If you've ever seen the, have you ever seen the pit? Yeah. And they have an example of this.

[00:08:12] They have to turn off all their EHRs and they have to, you know, figure out what to do when this happens. And it's a nightmare and patients are routed to other hospitals hours away. And it's a challenge. Yeah. The public information about Project Glasswing, what was happening is they were finding vulnerabilities that were never before understood. Deep in the roots of tech stacks that were not previously discovered.

[00:08:41] And that caused them to keep Mythos from being released to the public. Except for until, so basically they had three months and now we're at the three month mark where Mythos, the new model, the Mythos grade model was released this week. Now, I'm going to tell you, I haven't fully tested everything or kicked the tires.

[00:09:05] But what I have been doing over the last several months is working to coordinate with the health sector and taking my own security research based on what I know about agents. And helping people to understand where the breakdowns in the process are. So, for example, if a company is saying we have a SOC 2 compliant product and we're an AI company, are they approving those claims?

[00:09:31] Or do they have these backdoors as they are offering enterprise grade offerings to health care? Those are the types of things. If you watch this space and stay tuned, I don't have the things published yet, but they will be published soon. And I'm very excited about some of the work that we're doing. So fascinating. Now, the Mythos report introduced the idea that a model can be the safest and the riskiest system a company has ever built.

[00:09:59] Right. And they weren't having restraint. They weren't going to put it out. And now they put it out. So how should health care leaders think about that paradox? And should patients have been included at a seat at this table before they made this choice? This is the eternal question. Should patients have been included? And the first thing I will say is absolutely everything that we do in the design of technology should start with a patient.

[00:10:26] And it never does. And when we think about the people who are going to be directly affected by risks or benefits, there are going to be real benefits from more powerful models coming out. But at the end of the day, we're the ones on the playing field. We're the ones in the arena who have to survive or who may be harmed as a result of these benefits and risks.

[00:10:53] And so the question of should patients be the starting point for partnership and design? Absolutely. Did that happen? No. What is also, I think, very telling and surprising with Mythos is that with all of these frontier models, we're not really coordinating with the health sector as much as we could be. And I say that after talking to a lot of experts in the field of healthcare cybersecurity,

[00:11:23] there needs to be that step first, too. And you'll notice with Project Glasswing, what happened was they brought all the big tech companies in, but they did not bring some of the most important cybersecurity experts in healthcare to that table, too. The health sector is equally out of the loop, and we should be in the loop. I hope that we will be soon. I know that's not very comforting. No, not at all. The end of the world is coming. No.

[00:11:52] But I mean, at the same time, we have to be conscientious. And it does seem like we need to be saying there's a leadership vacuum, clearly, that each of the sectors are not being included. And healthcare in particular, where literally lives are at stake, you would think that they would include at least healthcare leaders, patients, caregivers, participants, folks that will be utilizing or potentially have been impacted by these models that they've developed and this entire program that they've put together. I want to talk now about Vibe Coding.

[00:12:21] It's making it possible for people to build applications by describing what they want in plain language. What concerns you the most? What excites you the most? What are you thinking about when you think about Vibe Coding? So I'm going to start with what excites me the most because I kind of get the reputation of being the Cassandra of health tech innovation. And I wasn't that way before. I do want to start with a lot of the positive here.

[00:12:48] We are unlocking ways for patients to build their own tools, build incredible and new ways to innovate that we should be able to protect. And I'll give a couple of examples here. I want to start with my colleague and our shared friend, Christine von Reisfeld. She at Stanford has been taking her data and not just with Stanford. Let me just be clear.

[00:13:17] She has been exercising her rights to data and putting it in local storage, thinking about ways that she can partner with experts and share her data on her own terms, but also thinking about what are the safe tools? What are the AI agents and Vibe Coding solutions that she can use in a safe and responsible way? There's also really cool stuff.

[00:13:43] If you've been watching patients use AI, I want to give a shout out to Hugo Compost. He took code from Josh Mandel, who's really well known as one of the architects of FHIR and exercising the rights of data, right of access to data. And he built his own tool called OpenKP, where he is pulling his records from the health system and then figuring out ways to visualize his own data.

[00:14:13] And I am so excited about these things because if you've been following the e-patient movement for a long time, or this group of incredible patients, of which I am one, how do I say that without sounding braggy? Literally not braggy. I am one too. I have been at the forefront of this as well, fighting for patient access rights. When you're advocating for something and fighting for something, girl, you can own that. I think all of us will say...

[00:14:40] I think as women, we don't own our power as much. And that's part of what this podcast is about. So as one of those awesome patient hackers who has also, for God, the better part of 13 years now, been fighting for the access to our data, we absolutely want to be able to do those things. And we want to be able to harness... I think the big unrealized thing that makes me excited is, it's not just about artificial intelligence.

[00:15:10] It's about patient intelligence and our lived experience. When we think about the things that never got designed for patients, I am now seeing some of these people in this space lead the way in building tools and doing so in ways that solve our own problems without needing to have a middleman speak for us. Let's cut off the middleman and just do things ourselves if we can do so.

[00:15:38] And if some of the places in healthcare that used to serve us are not adding value or access to what we need. So that makes me really excited. Now, there's a balance to that, which is, I think, when we look at the state of what I just shared about healthcare, cybersecurity, and where tech companies really have no duty of care. And I'm going to paint a picture for you.

[00:16:05] I want you to imagine this two sides, two cliffs, and in the middle is a rickety drawbridge where patients all live. And on the one side, we have the health systems, and they are moving at the speed of trust, a highly regulated industry. They've got to worry about all the things that are regulated, and they worry about getting sued all the time. On the other side, we have the tech companies that are moving quickly into healthcare.

[00:16:33] This year alone, Anthropic, Perplexity, and OpenAI have all announced their healthcare offerings. Many of them are pulling or asking, what is a way we can pull the medical records and chop and slice and dice it up with other data? And so here we are in the middle, trying to innovate, trying to survive, trying to just get basic access to knowledge. And neither side is actually supporting us on solid ground.

[00:17:01] They're letting us fall through the cracks where it's really more about disrupting each other. I like to say, this is what the current state of patient in the center of healthcare is. We get up on these stages and say, we're patient-centered, but we don't actually mean it. We pay lip service to it without supporting the infrastructure or the value that patient experience can actually bring to the design of technology. So now we've heard a lot about vibe coding.

[00:17:31] Are you doing some vibe coding of your own, Andrea? I'm hoping that the answer is yes. I absolutely am, but just not in the way that you might think. So a lot of people are rushing to put their medical records into OpenAI. And I have certain feelings and everybody has these different levels of comfort based on what they know about privacy policies.

[00:17:55] So instead of doing that, what I am doing is kicking the tires and vibe coding tools to be able to compare privacy policies apples to apples. Just because when you think about getting or signing on to a new app, there are all of these high-minded promises, right? But what do they actually say in the fine print? And what are their actual practices?

[00:18:19] So what I have been doing is playing around with how to vibe code public data and curate it in a way that patients can understand a little bit more easily than the legalese that is already out there versus what the marketing promises are. And there are some interesting things. I want to start with...

[00:18:43] There's a lot of these companies out here and in my little dashboard that I vibe coded have seen that good 37% that data point is going to change of the tools that we're using. Do you have in their privacy policies, they're going to target ads with Metapixel. Some of them do target ads, but they don't disclose it.

[00:19:06] So there are all of these gaps between what the marketing says versus the fine print versus what the app actually does. And I think it needs to be more transparent to patients. That's the way I've been using things. I've been thinking about how vibe coding can be a tool for education, a tool for learning to code. And instead of taking my deeply private data that I'm not going to put on one of these platforms yet...

[00:19:35] And let me just say, I really am a fan of local Claude. If I were to do it and kids make good choices, I would get a clean machine and do it with Claude. Follow Hugo Compost for some of the cool things he's doing. But for me, vibe coding is really more like there are other ways to compare public data and create dashboards that create value and use for patient communities and maybe change some of these practices in the market at the same time. That's one thought.

[00:20:05] And it also gives me hope. When we're looking ahead and we're thinking about AI, what do you think people are talking too much about? What is one trend we're not really paying enough attention to? And in a few years, what do you hope we're talking about? This is the hardest question. In a few years, if I can wave a magic wand and think about what I hope we're talking about,

[00:20:28] it would be that patients are self-governing the technology and data that belongs to them. And instead of these high-minded marketing claims on a lot of the companies' websites that are emerging in this space or the talking points we get from a lot of the leadership,

[00:20:49] I hope that we are actually making it possible for patients to build the future of AI and doing so in a way that isn't paternalistic, but is also protecting from some of those dangerous ways that I have unfortunately learned we can weaponize these vulnerable communities.

[00:21:11] One trend that I don't think we're talking about enough is how easy and dangerous it is when a community can be targeted. So let me explain what I mean by that. The Light Collective, when we published our research in 2022 around how ad tech gets used on the internet, often when you hear about patients being vulnerable to misinformation, we're like, oh, the patient is susceptible to misinformation because they're dumb.

[00:21:42] What we're not doing is talking about all of the research to show how the tech companies design these tools to be addictive, but also design ad targeting and marketing tools to follow patients across the internet. When we create a digital footprint around patient communities, then we start letting any scammer or snake oil salesman just blast them with content.

[00:22:06] And we wonder why certain vulnerable communities got radicalized or lost trust. I don't think we're talking about that. We just admire the problem of misinformation without looking under the hood at what is happening with patient communities who lack access to healthcare. And yet, when we turn to the internet, it's often unsafe spaces that have these 50-page privacy policies that write off any duty of care or harm.

[00:22:36] I think it's very interesting the role that you sit in the middle of patient rights, what's going on in the AI world, what's going on in the interoperability world, the infrastructure, healthcare data infrastructure world, the cybersecurity world, healthcare world. You sit in the middle of a lot of things happening. What is something that's giving you hope that things are going to change? Is there something that you're holding on to here? Yeah.

[00:23:00] I keep coming back to the true north of lived experience and valuing our lived experience. When patient communities come together and support each other, when patient communities really understand how to get ahead of the dangers, while picking up these tools to build amazing things. And I'm not by any means saying we should not partner with experts.

[00:23:28] We absolutely need to work closely with, depending on what community you're in, the experts in cardiology and cancer research. These are people who are picking up the tools too. And the one thing that really gives me hope is that we're all trying to figure out how to use AI. And it's an exciting and terrifying time. Let's lean into our communities, our lived experience in a way that supports each other,

[00:23:57] that builds on our shared knowledge, that helps us realize our potential in ways that are true to that human experience that we have when we're trying to heal from something traumatic or the difficult parts of finding out that you have a catastrophic diagnosis. Patients have always, when you become an advocate, turned that into something good. That's what you're doing.

[00:24:24] That's what, as a patient advocate, you're doing with this podcast. We all have these missions. I mean, we all see ways that we want to do good things for the world and innovate. And that is the thing that gives me hope. Thank you so much. Thank you so much for fighting for patience with the Light Collective and all the work that you're doing over there. It's just your advice and what you've taught us today is so valuable. To finish this conversation off right, where can our listeners find you online? If you go to lightcollective.org,

[00:24:54] I would actually invite people to join the Light Collective, join our movement, and help us shape the vision for how we get ahead of some of these questions in AI, how we learn together, recognizing we don't all have the answers, right? We just know that we're coming from the right starting point. That's what's most important. Too good. Thanks so much for joining us today. Thank you. And thanks to you guys for joining us too.

[00:25:20] Check out the Hit Like a Girl pod and Hi-T with Grace website for more interviews with great guests like Andrea today. Cheers! Thanks so much for listening to this episode of HIT Like a Girl Pod. If this episode resonated with you, please follow us on social media at HitLikeAGirlPod. That's H-I-T-L-E-G-I-R-L-P-O-D across all platforms. And to learn more about the show, our guests, or to join our waitlist for what's coming next,

[00:25:49] please visit our website. The link is in the show notes. The Hit Like a Girl podcast is built on joy, fueled by love, and led with grace. We're so glad that you're here. We'll see you next time.