Provide a summary of the following web Content, written in the voice of the original author. If there is anything controversial please highlight the controversy. If there is something surprising, unique, or clever, please highlight that as well. Content: Title: Can GPT-4 *Actually* Write Code? Site: tylerglaiel.substack.com Since ChatGPT came out I’ve seen quite a lot of people posting about its capability to write code. People have posted about how they had it design and implement a number puzzle game (without realizing that that game it “invented” already exists ), how they’ve had it clone pong , and hell I’ve even used it to write a few simple python utility scripts . It’s very capable, and a quite useful tool. But, there is a commonality in all of these examples people have been posting. They are all problems that have been solved before, or extremely minor modifications to those problems. And while, to be fair, a *lot* of programming is literally just that- gluing together existing solutions and fitting existing code into your specific use case, the *hard* part of programming is solving problems that haven’t been solved before. So I decided to test it on a particularly difficult “algorithmic” problem I solved a couple years ago. Something small and isolated enough that I could fit it into a ChatGPT prompt, but with enough subtleties that I feel like it would have trouble getting it right. Let’s start with a description of the actual real-use-case problem. In Mewgenics , movement abilities use pathfinding to get the cat from his origin to his destination. Cats have a maximum movement range stat (in this case it’s 6) and tiles have a cost (in this case its 1 for basic tiles and 9999 for blocking obstacles). We also have water tiles that cost 2 to path through. So far, nothing I’ve described is out of the ordinary for grid based pathfinding. Both Dijkstra’s Algorithm and A* can handle tiles with different costs just fine and its trivial to just cut off the path after a certain distance to deal with the maximum movement range. The complication comes in when we add Fire (and other hazard type tiles) into the mix. Fire tiles don’t cost extra to pathfind through like water tiles do, however a cat really wants to avoid pathing through the fire tile if it can. Shown here is the working solution to that. There’s a lot of more complicated cases as well, if there’s a lot of fire tiles it should go through the least number of them it can and such, and that’s all working well. Now, its not completely obvious why this is a difficult problem. Or at least, why the solution is more complicated than simply A* with a modified heuristic that considers “desire costs” like fire. It’s a problem that is very subtly different to the standard A* pathfinding, and the solution, while very very close to A*, has a few non-intuitive changes that mattered a lot. This was a “hard” problem, and back in 2020 when I was first adding fire tiles into the game, this pathfinding problem took a couple of days to actually properly solve. The main complication is the limiting factor of “maximum movement range” and how it interacts with both the cost to path through tiles and the “desire cost” used to avoid hazard tiles. You can see in that gif I posted how the path changes as you try to move farther and farther past the fire tile, eventually pathing around it would not be possible within 6 movement, so it has to path through it instead. And there’s a lot of complication to it beyond the simple cases too: (14 total movement vs 10 total movement on the same level) There’s a sort of built in assumption in A* and Dijkstras that, if the shortest path from A to C passes through tile B, that it also takes the shortest path from A to B. So if you’ve found the shortest path to B already, you can just start there and continue on to C. Those algorithms rely on that to be efficient, because you can skip tiles that you’ve already found a shorter path to when pulling stuff of the priority queue, and when reconstructing the path at the end you can rely on this by having each tile store the tile it was reached from during the pathfinding algorithm This is what I mean: This above situation does not happen with “standard” pathfinding! The best path from A to B does not line up with the best path from A to C, despite the path from A to C containing tile B. This uh, complicates things and breaks some assumptions that existing pathfinding relies on. So, its A* or Dijkstra with modified costs and heuristics right? Well… almost, but not quite. You can see the code for it below (along with some extra stuff for minimizing bends in the path as well). You’ll notice… it’s not exactly dijkstra or A*, and there’s a lot of stuff in there that is not obvious why its the way it is. In fact, I wrote this in 2020 and have since cleared it from my mind… it’s not even obvious to me anymore why stuff is the way it is. There was a lot of trial and error to get this to work, a lot of moving things around and doing slight modifications to the algorithm. In preparation for this blog post I double checked and tried to run it back and make this just modified A* again, but every change to simplify it just added in a bunch of pathfinding bugs. So its the way it is. And the train of thought that has led here has mostly been dumped from my mind, since it works already. Why is the path stored inside the cells that get added to the priority queue, instead of the standard way A* does it where the nodes store which node they were reached from? Why does it check that the path is the best path when popping from the queue instead of when pushing to the queue? Both of these are decently large diversions from A*, but absolutely necessary for this to work apparently. This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Anyway. That’s all the background info for the problem. You’re welcome to try solving it yourself if you want to get a real feel for just how not-trivial it is (or if you think that it really could just be A* with a modified heuristic instead) (and if you are going to try and “suggest” to me ways to improve it, then please actually implement and try it in context first, and know I won’t use it because what I have works fine already, has been *thoroughly* tested on thousands of different cases, and doesn’t have any performance issues either) Here’s the meat of this article. The actual experiment. Can GPT-4 (via ChatGPT Plus) actually solve this problem? I asked it Write a function in code that solves the following problem: An agent needs to find the best path on a 10x10 tile grid from their current location to a target location. They have a limited movement range of 5 points Regular tiles cost 1 point to move through, water tiles cost 2 points to move through. Fire tiles cost 1 point to move through, but they should avoid pathing through them even if it means taking a longer path to their destination (provided the path is still within their limited movement range) For the purposes of the experiment, I explicitly left out details of the algorithm, instead gave it a description of the *problem* that needs to be solved. This is how you start with a problem anyway, and I *am* actually curious if there’s maybe a more obvious or trivial solution I missed, so I’d rather not pollute it’s train of thought with implementation details. I also didn’t say what language to code in since I assumed it will work better if it picks python rather than asking it to write C++. I only care about the actual algorithm after all. ChatGPT responded, You can use the A* pathfinding algorithm to find the optimal path considering the costs and constraints. Here's a Python function that solves the problem: This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Which like, it *sorta* understood the problem, though instead of “avoiding” going through fire it just… never goes through fire ever. So I ask this code refuses to path through fire, it should still path through fire if it has to in order to reach its destination within its movement range And it apologized and then fixed the a_star_search function like so This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Again, not quite. It made no distinction between the penalty cost for pathing through fire, and the physical cost for pathing through tiles, so it still wouldn’t path through fire (as the +1000 penalty makes fire tiles way out of your movement range). Anyway I went back and forth with it a few more times, had it generate a test case, and fixed a runtime error that occurred when I actually tried to run it. Landed on this, and I customized the test case to something simple that would fail pretty obviously. It mixed up x and y in its test case stuff, but I didn’t want to argue with it over that so I just dealt with it (its pretty slow to re-generate) This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters On the test case given, because it ends on a fire tile it just ignores ever trying to avoid the fire tiles. I asked it to fix it, and it gave me a solution that went back to just failing to pathfind through fire tiles at all. I asked it to fix that, and it went in a circle back to the one it had before. After going in circles a few more times, I decided that was it. It got *close*. It seemed to understand the problem, but it could not actually properly solve it. Would this have helped me back in 2020? Probably not. I tried to take its solution and use my mushy human brain to modify it into something that actually worked, but the path it was going down was not quite correct, so there was no salvaging it. Again, the modifications from basic A* to the actual solution are not obvious, so starting from a base of “its just A*” isn’t really that much of a help. Moreover, GPT doesn’t even really recognize that this problem is harder than just “modifying A*”, an insight that would have saved me time when I originally wrote this algorithm. Anyway, perhaps this is not a problem suited towards GPT. After all, A* is an extremely common algorithm, one which it certainly has thousands or millions of examples of in its training data, so it may be restricted from deviating too far from that, no matter how much you try to push it and help it. I tried this again with a couple of other “difficult” algorithms I’ve written, and its the same thing pretty much every time. It will often just propose solutions to similar problems and miss the subtleties that make your problem different, and after a few revisions it will often just fall apart. In the case of this one (this is how Mew resolves Knockback) it kind of just missed *all* the details of the problem, like that it needs to not let 2 moving objects move onto the same time. This one was particularly bad, in that whe