Dario Amodei el 2016: cinc problemes concrets de seguretat IA

So as Bill mentioned, I'm a researcher on the Google Brain Team and I'm also an advisor to the OpenFland Theory Project, which thinks about social impacts about AI. So I mostly in the past have worked on issues in applying deep neural networks to speech and vision, but recently I've become very interested in the issue of safety and reliability of AI systems. I got together with a bunch of collaborator, something Google, some from Stanford, Berkeley, or OpenAI in the last few months. And we recently released this paper called Concrete Problems in AI Safety, which tries to bring some kind of categorize and bring some precision to the AI Safety discussion. So that's what I'm talking about here. And so pretty much me and all the collaborators on this project ultimately got interested in this because of our experience working with. deep neural networks which we found over the last few years to become very powerful and capable, but also they can be very opaque and can behave in very unpredictable ways. All of us kind of had the same thought that as these systems continue to become more powerful and be applied in more kind of safety critical states, this is going to be coming increasingly important area.

We kind of define in the paper the term accidents as what it is that we're worried about, which I think is the same as whatever we're talking about today, unintended and harmful behavior and machine learning system. And I am my co-authors, kind of we mentioned the paper that there are some trends that we think are really going to raise the importance of accidents. And they've been mentioned by some previous speakers today. But first of all, reinforcement learning, which allows increasing intertwinedment between the agent and its environment, all the progress on Atari and on GO. Complex agents and environments, which just increase the range of strategy the agents can have and the way the environment can be disrupted, increasing autonomy, which can take the humans out of the loop and end-to-end systems, which are often maybe a system might made up of many deep learning components that talk to each other and it makes it more difficult for a human to understand what's going on and to intervene in the system. in some way, in some way that's helpful. So we really, with these trends, we really wanted to take kind of a forward-looking approach to how to think about the safety of AI systems. And there has been some public discussion of forward-looking approaches, but I think a lot of it has focused somewhat, unhelpfully, on these kind of very extreme scenarios like AGI or superintelligence. And so we wanted to take an approach that was, yes, forward-looking, but also concrete empirical could be worked on now

could relate to systems that, you know, that, that we deal with today. So one of the, one of kind of the main things we do, we do in the paper is try and provide a categorization of the ways that accidents could happen. And actually, I think you could have many categorizations, but, you know, we're, we're just kind of trying to provide one framework among many for, for thinking about things. So what happens when a machine learning system does something you didn't want us to do? One way to describe it is that the designer had in mind a certain informal notion of the objective or task that it was supposed to accomplish. The actual system failed at this catastrophically. So throughout the paper, we use the example of a cleaning robot whose task is to clean a room. So my informal objective is it should clean the room in a way that when I look at the room at the end, I should be happy with what it's done. And if the robot, you know, like damages property, your injuries, someone or something like that, then, you know, that was not what I had in mind in my informal objective. So we can kind of classify where did things go wrong? So it could be that you wrote down the incorrect objective functions. You had an informal objective function in your mind. You turned that into a formal objective function. And no matter how perfect or unniscient your system was at learning and inferring things, it's just the wrong objective function and it will always do the wrong thing.

A second case is where the objective function is expensive. We might know the true objective function. For instance, we could always consult humans and ask them to do a detailed evaluation of whether every single action taken by the agent is a correct one, but it's not practical to do that. So we end up using the objective function only. You know, only occasionally and, you know, as a result, we can end up doing the wrong thing. And the third class is you have the right objective function, but something goes wrong during the learning or inference process. There's kind of a fourth category, which I don't talk about, but which I think is very important. It's just kind of different topic from what I'm talking about, which is non-ML problems that can arise as part of larger ML or AI systems, software implementations, verification issues, security issues, that sort of stuff. So I'm kind of going to go through the the three types and because this is a very brief talk, I'll probably probably go through my slides pretty quickly, but just want to give a flavor of what it is that we're talking about. Most of this is posed mostly, but not completely in terms of reinforcement learning agents because we see that as kind of a good paradigm, but a lot of this stuff we imagine might apply more broadly. Under the heading of wrong objective functions, we think about two ways

two broad ways that you can easily get the wrong objective function when you meant to write down the right objective function. One of those ways is what we call side effects, which is that systems that narrowly focus on one aspect of an environment that could be very large in the real world may end up disrupting other parts of the environment by default. I'll explain a bit a bit more later, but we have this running example of the cleaning robot. The idea is that we incentivize the cleaning robot to move a box And if there's a vase in its path, and we haven't thought carefully about the vase, it'll just knock over the vase while it's moving the box, and we probably didn't want that. Reward hacking has been mentioned in some of the previous talks, this occurs when an objective function can be gained, where there's a cheap way to achieve the formal objective function without achieving the ultimate intent of the informal objective function. So, sample is if we choose to reward our cleaning robot for not seeing dirt, it might close its eyes instead of cleaning, because it just sees no evil. Here's no evil. There's no dirt. So a little bit more on side effects to think a little bit more formally about it, we might have an environment where we're trying to do something like incentivize this to move the box. So we might naively write down an objective function that maximizes something that refers to only this variable in the environment. And when we do that, we're implicitly saying that we don't care about this other variable in the environment, namely,

namely the vase. What we really wanted was some complex objective function, which I'm calling F aware, which says like move the box while respecting some common sense constraints about like the other parts of the environment that are like really hard for me to write down. And you know, in this particular case, while I can just take into account the vase, but you know, what if there are 10,000 of these variables? And in particular, what if the variables I don't care about have more commonality across goals and agents than the variables that I do care about? That suggests that maybe I can take an integrated approach to this. So here are just quickly a few of the ideas. One is what we call learning a side effect regularizer. We can think of this as maybe a transfer learning problem where I learn in one environment and one thing I learn is how to accomplish the goal and the other thing I learn is how not to disrupt the environment. And maybe the information about how not to disrupt the environment is more transferable. It could be a separate neural net or something else module that can be applied to other environments without much or any change even when I have to entirely relearn the goal. Just as we do for like in model-based reinforcement learning, when we repurpose the state

state transition model, but maybe not the value function. There are information theoretic measures. A little bit, a little bit of a little bit of a skewer of it's starting to be used more in RL that measure in intrinsic measures of reward. And we've wondered whether those measures like empowerment, which measure the agent's theoretical ability to change its environment could be relevant here. We could take a kind of Bayesian approach where we think of the reward function as something that hasn't certainty in it and we attempt to learn it. We could use Bayesian methods and then kind of this thing about side effects is might be something that relates to the prior. Another thing is multi-agent approaches, which I think there's been a fair amount of literature on it has been discussed here. Cooperative Reen, Inverse Reen, Forcement Learning was mentioned once where an agent and a human work together to optimize the human's values. Mechanism design approaches where instead of thinking of side effects, we think of other agents that care about the environment and committing negative externalities against them. So reward hacking, here are a few things for reward hacking, just a few ways that you can kind of get the wrong objective function or an objective function that's gameable. There's something called good hearts loss, which I just found out when I was reading about this, which is when a metric becomes a target, it ceases to become a good metric.

So, for example, in the example of our cleaning robot, if you notice that we could, you know, that how good a job it's doing of cleaning is proportional to how much detergent it uses. And this correlation holds almost perfectly. You could imagine rewarding it that way. And then the agent may respond by dumping out all of its detergent and not cleaning anything. And this only happens when you optimize on that goal. Partial observation is a big one. Goals and rewards in the real world aren't experienced directly. which there are soon to be experienced directly in kind of the formalism of reinforcement learning. And so there's all kinds of aliasing things that can happen. Again, example of the robot closing its eyes so that it doesn't see any dirt. And if its reward function isn't constructed carefully, then it will think that its job is done. Wireheading has been discussed by Stuart Russell and some people in the Isdia group, where reward signal is implemented in the environment and so you can physically tamper with it and replace its own reward function. So this is kind of like self self-modification aspect to it feedback loops where there's a piece of the reward function that can amplify itself and drown out all other pieces of the reward function. So Leon Beto gives an example of font size and ad placement where ads that are more successful you display in larger fonts because you think people want to see them more. But this is a this is a self-fulfilling feedback loop and so you have these giant ads that may not have been popular at all.

and simply complicated systems. Any system that's complicated just like with software, it's increasingly likely to break under optimization pressure. Someone showed that with tool assisted speedruns, you can execute arbitrary code by just pressing the control buttons in Super Mario. I don't think our allergian has ever done that, but the environment is such that they could. So some approaches to it, something that's been mentioned a lot by Eric and a couple other people. We shouldn't underestimate this verification model. a monitoring isolation, unit testing of parts. There's this idea of adversarial reward functions, where maybe the reason this happiness happening is that the objective function is this kind of static thing, whereas the agent is this active thing that's trying to like, you know, it's trying to outsmart the reward function in some sense. So is there some way we didn't get super specific about this? that you could use generative adversarial networks or other unsupervised techniques to try and make the objective function whatever it is more, more resistant. Making your reward dependent on your model of the environment instead of something that's given to you is something that can help deal with partial observation and isolating yourself algorithmically from parts of the environment that relate to the reward. Okay.

Expensive objective functions. So here we only have one problem and it's basically the problem that I described how do we ensure safe behavior when the true objective function is too expensive to evaluate frequently. So human could tell you what the right thing to do is but you defeat the purpose of automation if you always have to ask a human. So the example is a clean robot might often find human possessions. It can't ask every time it finds something whether it belongs to a human. It might end up throwing out the wrong one so it has to learn. when to ask and when it should know based on what it's asking. So I won't go into this in too much detail, but one kind of toy system we start playing with and actually have started doing experiments with is this set up of a semi-supervised or active RL, which is the idea that there's a reward. It's just like RL, except there's a reward in the environment, which we can think of as maybe coming from from human supervision, where the agent actually has to actively request the true objective, and can only do so every once in a while. And so can it do the best it can to kind of like to learn to do the right thing based on that very limited information. And so one other thing we could we could study, and I'm skipping over a lot of stuff, is we'll be agent over time as it requests the true objective, begin to learn

Learnable proxies, say instead of a human giving a detailed evaluation, it gives the human a button that it can say yes or no to something and get a much quicker evaluation. So, will the agent learn to develop learnable proxies that are reliable and that are ultimately tethered to the true objective? So, all these kind of variations on reinforcement learning where the supervision is weakened in some way or altered in some way seem like they have a fair amount of promise. Correct, correct objective function problem in learning. This is a case where the agent, if it kind of had infinite time and correct beliefs, would do the right thing, but something goes wrong in the meantime. So safe exploration is how do we avoid catastrophic exploratory moves? We want a robot to experiment with clean strategies we don't want it to stick a wet mop and electrical socket, we don't want helicopters to crash as one of the previous speakers discussed. Distribution will shift. which I think has also been discussed previously. How do we detect and respond gracefully to differences in test distribution? Oh yeah, sorry, that example is wrong. The example I meant to give is like the robots trained a factory for it, like uses a bunch of harsh chemicals and like then it works in the office and that that doesn't make sense. So generally these have gotten a lot of attention and so those those sections of our paper are more kind of reviews but also with pointers to how this can be applied to kind of emerging.

emerging deep learning applications. So just going a little bit quickly through it, there's substantial prior work in this area. I think there's a lot on adapting risk sensitive performance metrics to deep learning, simulated exploration, especially with the advent of model-based reinforcement learning in deep RL setups and setups that you've bounded exploration to ensure air to ensure air-gadicity. ML systems can fail silently and unpredictably when the test distribution differs from the train distribution. This is a room that like eight someone's here. With classifiers, I mean in some sense things can't be very bad, but for people who remember this, a classifier can do something incredibly offensive if it's not trained in the right way, you give it the wrong inputs. RL agents can act on really bad. perceptual info, we can add ad hoc restrictions to make sure that we kind of detect this, but I think ultimately we need and should be able to have a machine learning solution because you know if you're outside the train distribution, you have enough information to detect that your inputs are different from where they were before. And there's been a lot of work on this. I'm running short on time, so I'll just summarize it very quickly, but kind of covariate shift assumptions that make assumptions about the probability of the output given the input being the same and only the inputs changes, and the converse of that,

There's, I think something very interesting is like broadening the train distribution and understanding if, if, if I train on a few different types of data, how much of that does it take to generalize, how to why determine when I've generalized well, and also how to respond in agents, which I think has been a very, very understudied topic. There's been a lot of study of like, how to detect being out of distribution, how to respond to it in classifiers, but agents can do a whole lot of things, right? If I, if I see a chemical that I've never seen before, You know, then I as a human can experiment with that chemical and the controlled environment instead of, you know, even if that's more expensive instead of doing something else, so I can try and access more training data. So I think there's a whole world of kind of agents remedying their own uncertainty and knowing what they don't know. This is some stuff we've kind of started proposing experiments which I won't kind of go through in great detail, but The thrust of a lot of these is kind of looking for large suites of simulated environments where we can have very broad setups where we can test agents in a way that they don't overfit and they can actually be shown to address these tasks. And just as simulated environments and video games have been very important for go, when Atari and reinforcement learning in general, my hope is they can be very important for safety. And in fact, within a couple days of publishing the paper, some folks that open AI implemented

a couple of the ideas we had in OpenAI, which is one of their tool open source tools for RL, just just for. And so finally, since there seems to have been a tradition that everyone mentioning a little bit, the long-term concerns I did want to give my perspective, which is I do think that kind of long-term concerns about AGI and superintelligence can be seen as kind of extreme versions of accident risk. I'm not from there taking, but kind of the more academic writers like like Nick Bostrum who talk about, you know, if you make a machine that makes paper clips for you, maybe makes only paper clips that it takes over the world. So this is kind of an extreme and more fanciful version of accident risks. I think it's too early to work on this directly. I don't know what such systems would look like and what to do with them. But what I do feel strongly is that if we take a principle than general approach to the safety of today's systems, that is the correct response to concerns about the far future. Andrewing, who was my former boss, had a quote, worrying about super intelligence today is like worrying about overpopulation on Mars. But I'd like to propose a bit of a modification to that, even though I think I agree with a lot of it, which is, it may not be worth worrying about overpopulation on Mars today. Can't ensure it study overpopulation on Earth. That is a serious issue we can think about. And if we do that right, then a lot of what we learn may someday apply to Mars, if and when we do go to Mars.