LLMs reward expertise

I did a test a few months ago. A friend of mine wanted to develop what i understood to be a simple single page web app. But since she didn’t have any software engineering experience she asked me to help. Around that time everyone was talking about how literally anyone can develop software with LLMs i asked her if she could give it a try first, and if I could watch the attempt.

I was fully expecting that writing the code will pose no problem for the AI. But i was curious if the AI will realise that my friend is a novice and needs extra help with things like: copy pasting the code into a text file and saving it with an html extension, helping her host the file online so she can share it with others, buying a domain for it, etc. I assumed they will get there eventually, but i also assumed that it will take a lot of stumbling around and misunderstandings.

But i was completely wrong. They didn’t even get to that point. Because my friend didn’t have the vocabulary to ask the AI to write code. They were just going around in circles where the AI was brainstorming with her about possible features and getting thints more and more complicated. We terminated the experiment after one and a half hours and many many messages exchanged between her and the LLM.

Whereas to me who knows the terminology would have probably just taken a single exchange of messages to get the result she described to me. I would have prompted with something like “Please write to me an html page which does X, Y, Z.” But since she didn’t know the right terminology she got into a vortex of feature discusion, and she didn’t find a way to tip the AI into “just do it, write it now” mode. In other words in that case the LLM would have rewarded even just a little bit of expertise, but without it there was a confusion about goals between the human and the machine.

The amplifying mirror analogy works best here. LLMs are ultimately a reflection of your own interactions with its weights, the tone you use, the structure with which you construct your prompt, aspects of an issue you tend to focus on, your breadth of vocabulary and world knowledge and whatnot.

People who (carefully) use it as an extension of their own mind and senses will very likely thrive, and those who use it as a replacement for their minds and their senses will struggle.

One of the Claude skills I made Claude itself generate was the 'learning a concept across tiers' skill -- from ELI5 level to a PhD level, and it triggers whenever I ask it a very general question on a complex topic that isn't my bread-and-butter. The fact that I'm able to choose explanation level from a super smart LLM (that's available 24x7) that can explain any topic under the sun would've been mind-bogglingly sci-fi-ish just 4 years ago in 2022.

Both the article and some of the discussions here share a lot of commonalities with doctors taking a medical history. There is a certain skill in guiding the conversation towards useful outputs, while not dictating the exact outputs to a patient who is eager to please with their responses.

E.g., medical history taking protocol always says to start with open ended (albeit structured) questions, and converge towards more closed/specific ones when you're sure you've extracted the broader surface and you now want to close in on a differential diagnosis.

If you start open and go with the flow but then just let the patient talk without any structure or subsequent attempt to converge, there's a risk that the patient might spend 60 minutes taking about their fluffy dog at home, which wastes time, and doesn't get you anywhere nearer the diagnosis. But, if you skip the open questions and go straight to yes/no diagnostic questions, you will definitely miss the fact that they have a dog at home that they're worried about, and that they'll be self-discharging against medical advice in the next hour to go tend to their dog.

So while to an outsider, the conversation might look effortless, in reality the doctor requires considerable skill to be able to strike a balance between open vs closed prompts, as well as the ability to critically sift through the outputs, and decide which outputs are relevant to pursue further and lead to a fruitful direction, versus those that can be safely discarded to remove potentially distracting noise from the conversation (and all while attempting to keep this interaction within a limited number of prompts due to operational time constraints).

This is something that really needs to be formally studied.

I'm inclined to say that this matches my own experience, but I can't rule out confirmation bias on my part.

As a meticulous person generally looking for a very specific code outcome, I prompt in a way intended to get exactly the thing I have in mind, and my results reflect that. But on the other hand, I have coworkers who type ten-word prompts with very limited specificity, and they seem to get results that way as well, and that makes me wonder.

It would certainly be beneficial for my career and financial well-being for the assertion to be true, because it means I don't have to worry about being pushed out of my job by an army of $15/hr vibe coders. But the convenience of that assumption is exactly why I think it's important to be skeptical.

> Of course both are useful, but I’d rather have familiarity with the codebase than a deep general understanding of software systems.

In my experience, getting that familiarity with a particular codebase in a way that isn't surface-level has always been a hands-on process. E.g. just because I know many general things about software, I need to know the particulars of the current codebase I'm in to know what is reasonable to actually apply to it.

This is a chicken and egg problem I find hard to resolve with LLMs. If we're pushed to delegate most work to them, how do you build that expertise? Sure you can ask questions about the codebase, but IMHO that falls under surface-level information, and the devil is often in the deeper details. Hmm.

This reminded me of Gaussian Processes. You start out with n-dimensional unconstrained (but strongly correlated) gaussians. As soon as constraints (data) are added (mathematically it's called conditioning), the thing goes more and more into shape.

Prompting feels a lot like this conditioning phase to me. You start with an LLM in unconstrained mode, basically just a "soup" of knowledge. If you prompt wisely, you immediately condition the LLM into "your space of (domain) knowledge".

What comes out is an extended version of your existing knowledge.

I am not sure. Why do you need domain expertise beyond being able to draft a verifier for the problem? Once you have the verifier, it's just a matter of compute. You can argue that being a domain expert allows you to narrow down the search space for the LLM and save time/compute costs. This is partially true, but LLMs are getting better and better at search (they can already do end-to-end performance optimization faster than performance experts at a FAANG company I work at), and compute to maintain the same intelligence level is getting cheaper.

A concrete example: GPU performance optimization for a kernel. This was (and still is) a very niche domain with not many top-notch experts. But kernel performance and characteristics are easily verifiable. You can run the agent in a closed loop for it to improve iteratively (and people are already doing it, coming up with kernels better than human-written ones).

You see Tao's example because:

1. He is curious (so he asks detailed questions, which are not necessarily needed in a closed-loop optimization).

2. Verification in math is harder. Many math tasks used in RL are easily verifiable. But for advanced open conjectures that require long proofs, you cannot trust the proof directly from the LLM (so it's not as easily verifiable as basic math problems or code). The model needs to write it in Lean, and you still need to make sure the Lean implementation correctly captures the specification of the problem. So you still need a human for verification in advanced math. But I don't see why you would need this in domains like performance improvement.

I do find that "signalling expertise" is important. "I have a significant background in biblical scholarship. You can assume I've read the most important works in NT studies in particular. Do not translate Greek, Latin, Hebrew, or Syriac. Now, I would like to know . . ." That changes things significantly. So does telling it you have 20+ years of experience with C programming, that you have a robust understanding of machine organization, memory layouts, embedded systems, etc.

Not sure I agree with this. The math guy at anthropic's prompts are essentially:

  "suppose you’ve gotta resolve the $CONJECTURE, like absolutely have to, everything depends on it. think really hard, and try to come up with a bunch of ideas to try. but remember to trust yourself and not necessarily in conventional wisdom!!"

  https://claude.ai/share/25740bd5-aa97-4bd7-bf58-c4df3793fda7
  https://xcancel.com/__alpoge__/status/2083855298239078748

What's funny is that every other person gets a different conclusion about who these models reward/empower. I've seen people say that the generalist stands to gain the most and others say that it's the experts. Like all of life, maybe the "winner" is the person who just does stuff.

The counterexample of the Dinitz-Garg-Goemans conjecture was basically just "keep going" and finally "enough of partial results. now finish with a complete unconditional counterexample" lol

https://x.com/DmitryRybin1/status/2079904005652893709

https://chatgpt.com/share/6a60b2eb-0b64-83ee-9c76-7931ca1de0...

The short version I give to non-technical people who ask me about whether "AI will replace coding" is this: it accelerates you. You can get much further much more quickly.

If you don't know where you're going or how to get there, or even if you're just not paying enough attention, it will get you very far in the wrong direction before you've realised.

> The model outputs are much more concise than when I try and talk to GPT-5.6 Sol about mathematics. By signalling expertise, Tao shunts the model into “talking-to-mathematicians” mode, not “explaining-to-amateurs” mode

I believe this works in two different ways.

First, information compression. The use of professional language helps describe problems more densely with minimal information loss/distortions. Verbose output by LLMs (e.g. ELI5) tend to incorporate local chat context, which can destabilize the context (e.g. out-of-topic, irrelevant nitpicking on writing style and wordings) and lead to faulty logic and even hallucination. LLMs are not good enough to look through all the noise, so, sometimes, it's helpful to refine the input data before performing actual tasks.

Second, boosting logical pattern-matching. Using professional language helps drive logical reasoning through simpler pattern-matching b/w texts. This is not about whether LLMs can reason or not; it's about how high-level reasoning is guided by preconception. Even humans tend to consume only textual surface of highly complicated theories (e.g. Adam Smith's "invisible hand"), and use them casually during conversation. It's similar for LLMs: if the conversation is conducted entirely in professional language, LLMs can easily incorporate external professional information into its reasoning. If the text is written in amateurish tongue, translating it into professional language can introduce errors and distortions.

So, yeah, keep your conversation professional, tidy and tight. A large volume of unprofessional text helps no one.

> In the 2010s, if you had technical gaps (say, you couldn’t write CSS), you had to either rely on a skilled colleague or just hope that the answer to your exact problem was out there on the internet.

You could read some general reference/guide/tutorial documentation on CSS, and then probably solve your problem (without searching for "how to center a div", or whatever your exact problem was, and copy&pasting the answer and moving on), also becoming more knowledgeable in the process.

The rest of the short blog post has some good points, but the first sentence sounds like it's targeted at the percentage of developers who did StackOverflow copy&paste to close Jira tickets, never becoming experts.

Delegating to LLM-ish AI is just a natural evolution of that. The question is whether they can still add value if kept in the loop.

The article author suggests that the answer is to be expert, and is addressing people who... "either rely on a skilled colleague or just hope that the answer to your exact problem was out there on the internet."

I don’t understand why this is such a revelation. Anyone who has listened to a good/great interview knows the skill of the interviewer plays a big part. To ask good questions, to understand what the other person is saying (AI or human) - that requires skill, expertise and patience.

Someone with less skill or expertise might still get good results, sure. It would just take longer and it wouldn’t be pretty

This is why the chat interface is ultimately not the best option for non-expert users, because they require the user to bring knowledge with them. You can call this the “query” method: you have to know what to ask to get the answer you want.

A real world example might be: I can find any movie DVD you want from our warehouse, but you need to tell me the name of it. Don’t know the name? Tough luck.

Contrast this with a “browse” interface: the options available are presented to you, and you can pick from them. Relevant contextual information is already on-site. The DVD store has shelves of potential movies you can rent, and you don’t need to know their names ahead of time.

The interfaces of future AI will be more browse oriented, with a query viewer available in the settings for advanced users.

I've been equating them to graphing calculators since the first LLM launched. It's an amazing tool if you know how to use it. If you don't know how to use it, it's still a tool, but you won't be doing anything amazing with it.

I totally see this. I just did 3 hours of bot sitting to put together some thrash loops that thrash our provisioning working flow for a BLE gadget we make. It was pretty straightforward and productive. But then, I have a lot of experience with BLE, and a quite a bit of experience with python and shell scripting. So I was able to guide the process through stages, do some intermediate testing, make some adjustments, and proceed. Domain experience made this really easy and straightforward. Me two junior engineers who have only superficial/high level knowledge of BLE and some of the other pieces, couldn't have done this as effectively.

Where my angst comes, is worrying that no one will ever get that experience anymore. They might have had some eventual success, who knows what monstrosity a much less guided LLM would have done, but experential learning may be mostly a thing of the past. And it creates a real tension between the person with experience and the person without.

The LLM is like the death star. If you don't know exactly where to point it, you will likely miss your target and have no/negative effect. The further away the target, the more accurate your firing solution needs to be. If all you need to do is add something like a dark mode theme to an existing product, this is probably a point blank shot in this metaphor. Building an entire codebase from zero, or even refactoring a legacy codebase into a new codebase, are lightyears away by comparison. You can still land the shot, but you need to deeply understand the metrology and astrodynamics.

The information system required to encode the aesthetic preferences needed to make a technology experience not suck is likely in excess of what any near-term solution will offer. Knowing when to say "no" is perhaps the most important skill here. You can't just say it arbitrarily either. You really have to mean it and be willing to fight other humans for it.

I think this is extremely true when it comes to prompting, but not only in this way. I would add that this also applies to an LLM's output on deep enough topics.

Anyone can point at a public GitHub repo and have an LLM write the documentation for it. Whether that documentation is good requires understanding that codebase.

There's no way around expertise unless we're talking about strictly mechanical tasks.

I do think LLMs are incredible at helping to build your expertise. You could point it at a codebase and say: "Explain how this API works" and interrogating the LLM until you get an explanation at exactly your level of understanding.

That's why when people like Pieter Levels tweet "I cancelled and then vibecoded 100% of my SaaS subscriptions", you need to take it with a huge grain of salt because you're not Pieter Levels, you cannot vibe code your SaaS subscriptions.

There's a growing and fascinating divide between people who see LLMs as more of a "bicycle for the mind" in the vein of Jobs vs those who see them as whollly supplanting the role of human intelligence. I can't help but wonder if these aren't primarily two human archetypes more than anything - the LLMs can be both and they erect a mirror of the human using them. Some humans really don't want deep individual expertise and intelligence to be the deciding factor because they don't identify with that. Others are completely the opposite. We really can't tell which will be more effective yet, because LLMs are very good in both modes. But most of the predictions currently are people executing on wishful thinking about what they hope will be the outcome.

LLMs are language models. So much of this can be reduced to a simple heuristic: If you can't think clearly, LLMs will not help you. If you don't know what you're asking for or how to express it precisely, it really should not be a surprise that the output is garbage. Failures of LLMs are more often failures of our own brain to consider the problem clearly. That's harder to admit than just blaming the AI.

It's important to note though, from the perspective of the LLM's objective function, that this is not a failure at all! LLMs are designed to match patterns. You give a jumbled mess of incoherent ideas, it will faithfully reproduce a token stream of incoherent ideas. It's only when the model output is subjected to the real world that it fails.

I've experienced this firsthand and 100% agree.

The more cracked you are in a domain, the more you can squeeze out of an LLM. If you already know what "good" looks like, you can steer it, call out its BS, and iterate way faster than someone who's using it to learn the domain itself.

Sometimes after a larger source draft with AI before I even touch the program I start to pick up on some of the holes in my prompts caused bugs/unintentional mechanics to be potentially woven in. Knowing what you want and how it should be made is half of it, but if you don't supply the bot with extra guardrails eg. don't modify the contents of x, because of y don't allow z. etc. They will do what you ask of them, usually less than you'd hope.

I haven’t experienced this - or maybe the training data for aviation is limited. But ask a an llm for aircraft performance / flight planning data and it’s scary how bad the advice /feed back is.

I am feeling this a lot lately. Getting the most out of agents seems to require being able to ask the right question. And how can you ask the right questions without deep domain expertise?

This matches my experience. Just Talk To It is the best method for working with LLMs if you're an expert.

I've seen this at work (as eng manager/lead/principal/whoevenknowsanymore) – all the big APIs give you stats. We see how much people burn in tokens and we know how much output they produce. There is a pretty strong inverse correlation between token burn and output.

The more tokens people burn, the less likely they are to produce a good outcome.

A good moment to remind everyone that if we took the promise for granted, that AI will in fact prevail and prompting is the one skill that will rule them all... we'll lose all domain experts in one generation.

It's less of "signaling expertise" and more about actually having said "expertise".

In my experience with LLMs it's not uncommon to be having a deep conversation about making pasta, only to be told, after asking for a sample recipe, to get a bucket of paint and a bag of concrete. Of course these hallucinations are way more subtle and easy to miss for someone who doesn't have deep domain knowledge.

When I use Claude to write code for my own projects, the code it generates is exactly the same code that I would have written had I done it by hand. If there is any deviation, I ask it to adjust but that is rare. At least 95% of the time, it's like it read my mind... Which is quite impressive when it outputs like 500+ lines from a single prompt and then it works straight away without any debugging necessary.

I don't even debug anymore on those projects. If Claude tries to add debugging logic in my code, I tell it not to and just provide additional information and it can usually find the solution faster that way.

This is when working on my own projects. When working on projects created by other people, it's a different story and I have to fight it constantly to stop it from implementing hacks and workarounds... It uses much more tokens to implement basic features. It's more work for both the AI agent and myself.

The project's existing code makes up most of the context so if the code is not great, you have to write long detailed prompts to set it on the right path. You have to make it clear that the existing code isn't good enough and your expectation is higher.

In this case, it usually gets better with more back-and-forth... At the beginning, it can't do anything because you keep pointing out a problem whenever it tries anything at all, but eventually, after a lot of criticism, it starts becoming more careful and adapting to your standards.

So yeah, even same person doing the prompting can lead to two very different experiences depending on who built the foundation.

So my conclusion is that the expertise comes from both the existing codebase and from the person doing the prompting... And TBH, I would say the codebase/foundation carries more weight than the person doing the prompting.

Pretty sure I could put an idiot on one of my codebases with Claude Code and they'd do a decent job.

Agree with this. LLMs multiply the human user's ability. More ability, more impact!

At work we call this implicit steering. To use webdev metaphor, if a non-technical person describes making a web page with a big block at the top and some things to click on and then my pictures below that, that will eventually get somewhere. Meanwhile, if you know industry jargon, you might describe a hero, with call to action buttons, and then below a 3x3 grid of images of my portfolio photos—that's likely going to generate something entirely different and likely richer. It can assume things about you (it doesn't think), it can ask you specific questions a web personal might know, it can infer domain context that is otherwise omitted with a basic conversation. Everyone wants to capitalize on corporate vibe coding but the tech literacy is hardly there, let alone more advanced topics.

This has been my experience. I’ve been working on an app (highly successfully) since February, with the help of an LLM (ChatGPT). It has not been used as an author. Rather, it’s been a “coding partner.” I’ve been the one that has submitted the work to VCS, and run the tests.

I’ve found the most utility in having it do “small stuff that I could do, myself, but it’s faster to have the LLM do it,” and in analyzing intractable bugs, like memory and threading problems. It’s really good at analyzing a bunch of code, and seeing a small typo that results in something like a strong reference.

In both these cases, my own expertise is vital. I’m asking it to act as a consultant; to give me advice and material to be integrated into a whole that I am architecting.

I guess part of it, is that I haven’t been able to completely “give in,” and wholly trust the LLM, like I hear many people do (profitably, I guess). I’m used to having my sleeves rolled up, and my hands in the dough. Catching some pretty severe mistakes, from time to time, has reinforced this perception, on my part. I wouldn’t catch these, if I didn’t know what I was doing.

The interesting thing is, most messages were ending with just one question of his. Examples:

- ..Does this polynomial map have any symmetry or other structure that makes this cancelation less miraculous?

- Given this structure can you see the non injectivity in a transparent way?

- ..But why is the jacobian from x u r to P Q R just a monomial?

- ..Is there a general theory of such twisted jacobians and do you have any sense why those particular dilation weights were used?

- Given this weight structure, why exactly is x given by a cubic equation from P,Q,R?

Also, looks like Terence Tao was doing lot of work and asking LLM to verify. This is inverse of the LLM trend, where LLM does the work and humans verify.

Every thing passes through the following stages:

1. AI cannot do something.

2. AI starts being able to do something, but one needs to prompt it carefully, so one needs to be an expert, see, we will always need human experts <--- this article is here

3. AI just one-shots it.

Why do people still need to say this for each and every task? It's just reliving the bitter lesson over and over again.

This why those RL env startups are able to charge frontier labs so much for their work. LLMs still generalize poorly outside of self-verifiable tasks like coding and math.

Labs have to compensate with post-training in RL env that embeds these expertise well, which is non-trivial both in terms of domain knowledge and technical expertise.

The breakthroughs are coming from simple prompts, some made by people with no math training:

https://www.newscientist.com/article/2580932-extremely-basic...

The referenced Terence Tao chat did not lead to new breakthroughs.

Im beginning to think buzzword bingo is a viable job interview method.

If you use the b words the LLM would be somewhat constrained to content that has them. If you don't use them it will use content that doesn't have them.

The company website (subject) will never become a static html document. If you request exactly what static html does it should probably point you to a wysiwyg website builder.

The actual "prompting trick" that dramatically improves your results is often to add just two or three words, like "use library foo", "write in <language>", "<bar> algorithm". To know which two or three words apply in your situation, you need a deep understanding of both the problem and the solution space.

Your prompt might look almost the same as the one from somebody with a good understanding of the requirements but no technical competency, plus maybe one or two sentences. Those one or two sentences dramatically change the results, and what those sentences are differs from prompt to prompt.

If you go back to Alan Turings' paper, its all about chastisement! honestly, last pages are all about postive and negative (child!) reenforcement - 1950's style (I do not condone ... etc). Simple as that. I conduct high level litgation in the courts, and win because of a good LLM, with a good version of me, keeping it in line! also crypto and cyber sec. Of course, child rearing, and dealing with former spouses is also very useful. The orginal paper 1950 https://tinyurl.com/yuszahpw (punish is mentioned six times). Just say-ing-like. TTFN.

I consider myself senior engineer. When talking with junior colleagues they often are surprised how little I care about some things and how much I care about others.

These internal "attention weights" are highly influential parameters of how I work with LLM.

E.g. when working with Rust I often hold strict control over structures and lifetimes. But when lately I've been doing token-based bind generation I didn't care about anything outside of high level patterns like RAII and ultimately - API ergonomics which was verified in consumer app.

I've been in position of porting real-code to vibe-code platform and seeing non-technical people prompt-stream (they were shared across accounts) I know why they engaged engineer to run this work.

Their efforts took 6 weeks, I ported app within 4 days and (to be honest with myself) without LLM I that'd be 3M+ work pre-LLM.

In short: I observed same effect as claimed.

You're basically playing the role of team lead to the LLM's junior dev.

A fascinating thing about the LLM/AI blogosphere and X is watching memetics in real time. An idea like this one propagates on the order of days until everyone that speaks publicly or in workplace meetings about AI is repeating it.

> The model outputs are much more concise than when I try and talk to GPT-5.6 Sol about mathematics. By signalling expertise, Tao shunts the model into “talking-to-mathematicians”

Maybe, but FWIW my first thought when I skimmed Tao's session was that he probably has a personal system prompt requesting this style.

E.g. even if you get it into "talking to an expert" mode I've found AI waffling through filler like "given your background in Linux kernel engineering, I'll skip the surface level and go straight to the technical meat". You do have to explicitly tell them if you don't want this.

> Because my friend didn’t have the vocabulary to ask the AI to write code

Is it possible that the effectiveness of an LLM user with respect to the expertise of the user is like a sigmoid function or at least a step function in that shape? That is, one has to know something like the basic concepts and the vocabulary to bootstrap a programming project, but one does not have to know too much to do lots of meaningful work, and then again one needs to be en expert to build something extraordinary. Since most of the work is somewhere middle, most of us mere mortals are still concerned or stressed out for the possibility that LLMs will squeeze out too many job opportunities.

Prompt an image or video generator without knowledge in photography or art skills and your results will look sloppy.

"The most important skill in the AI era may not be prompting. It may be learning how to solve problems using the right kind of help." [1]

Context - I have over 25+ years in software, and I have this observation - being introduced to a new codebase as a human is difficult, especially depending on the scale/size and complexity of it.

Yes, you do start to learn it as you work through it, but if the scale is truly huge, it may just not be possible to fully read and understand all the code and paths etc.

I have found systems-thinkers (I believe I am one, sometimes they are architects) to be able to kind of "see the whole picture" while not knowing all the details, to the point of being able to guess how the system/software should be behaving, even if it is not actually yet. This is a hugely valuable skill and I think takes a certain kind of brain too.

That said, I think recently I may have realized something - we rely on statistics and confidence levels in order to make statements about larger populations. If we can represent a codebase as a, perhaps stratified population of code, interfaces, docs, etc. etc. etc. we may be able to take a valid random sample, review portions of the code, and make some kind of assertions about the state of the larger system - potentially, from that.

I am trying to implement this as a side project right now to see if there is anything to it, basically, a combination of AI/LLM + stats/sampling + facilitated expert human review.

I'd be interested to know if anybody is doing anything similar.

[1] https://www.actinginbalance.com/p/the-right-tool-rule

They may reward it but they don't build it, and therein lies the paradox

I actually don't feel like Tao's recently published conversation is the best example of this idea. As intelligent as Dr. Tao is, and surely more so than me, I got the feeling that he wasn't running up against failure states of the model, which I'm not sure you could attribute entirely to his expertise. I honestly think it was more a matter of luck that the model apparently had so much training data on the topic or that it was architecturally so well suited for it. On the other hand, I've had really surprising moments where Claude was just failing terribly to execute simple dev ops tasks having to do with log processing. And I'd be so bold to say that I don't think it could have been explained by a lack of expertise on my part, or even a misuse of the model.

So yeah, sometimes LLMs reward expertise, sometimes they don't. I guess either way it helps to have it.

This is an interesting problem. A similar approach that worked for us was to only load the tools the agent actually needs for each task.

I like it. And if you don't know a codebase, like normal, spend some time learning in. Use the LLM to query it, create your own architecture diagrams. Get homey with it before making sweeping changes. Maybe make the first simple changes by hand.