logoalt Hacker News

badsectoraculatoday at 11:40 AM2 repliesview on HN

The information about finetuning is interesting (it is something i'd like to do myself at some point, though i'll wait until i can do it with local hardware :-P). However FWIW LLMs are generally good at following a specific style when given examples.

As an example i asked Devstral Small 2 to write some docs for my LIL scripting language in the following style (this is copied from the DirectDraw documentation, edited to be text friendly):

    IDirectDraw7::CreateClipper
    ---------------------------

    The IDirectDraw7::CreateClipper method creates a DirectDrawClipper object. 

        HRESULT CreateClipper(
        DWORD dwFlags,
        LPDIRECTDRAWCLIPPER FAR *lplpDDClipper,
        IUnknown FAR *pUnkOuter
        );

    Parameters

    * dwFlags - Currently not used and must be set to 0. 
    
    * lplpDDClipper - Address of a variable to be set to a valid
        IDirectDrawClipper interface pointer if the call succeeds.
        
    * pUnkOuter - Allows for future compatibility with COM aggregation features.
        Presently, however, this method returns an error if this parameter is
        anything but NULL. 
        
    Return Values

    If the method succeeds, the return value is DD_OK.

    If it fails, the method can return one of the following error values: 

    * DDERR_INVALIDOBJECT
    * DDERR_INVALIDPARAMS
    * DDERR_NOCOOPERATIVELEVELSET
    * DDERR_OUTOFMEMORY

    Remarks

    The DirectDrawClipper object can be attached to a DirectDrawSurface and used
    during IDirectDrawSurface7::Blt, IDirectDrawSurface7::BltBatch, and
    IDirectDrawSurface7::UpdateOverlay operations.

    To create a DirectDrawClipper object that is not owned by a specific
    DirectDraw object, use the DirectDrawCreateClipper function.

    Requirements 

    Windows NT/2000: Requires Windows 2000.
    Windows 95/98: Requires Windows 98.
    Header: Declared in ddraw.h. 

    See Also

    IDirectDrawSurface7::GetClipper, IDirectDrawSurface7::SetClipper
And it did a fine job. I put the full transcript in[0] to check out. The neat bit is that it can even handle weird formats like a custom documentation format i have (which only exists in my PC because i haven't released it anywhere) for a "master document" that can then be converted to various other file types. I gave it an example of some code in that and asked it to convert the documentation to it (this is part of the transcript at the end). Then i copy/pasted the generated code to a new file (adding a few extra lines the doc system expects which weren't part of the example - BTW i did not had to modify the generated code at all) and from that i generated a CHM file[1]. FWIW here is a comparison with the DirectX page i copied[2] (though consider that the generated pages went through the doc format which forces its own style and the textual output in the transcript matches the given style better).

[0] https://app.filen.io/#/d/9f4c1225-3527-4f16-a522-0678342120c...

[1] http://runtimeterror.com/pages/iv/images/45f8df428afe4fe6b6a...

[2] http://runtimeterror.com/pages/iv/images/ee58032790a049d7e74...


Replies

zahlmantoday at 4:18 PM

> However FWIW LLMs are generally good at following a specific style when given examples.

In your experience, is it worthwhile to have an agent create a "skill" for itself for following the style? Or is it a better use of context to just have it review the examples?

theletterftoday at 11:44 AM

Interesting! I think the advantage of style fine-tuning is that you might not have to provides that much context upfront. Also, it's kind of magical to have an LLM just do something out of the box. I'll compare my local fine-tuned models against the baseline with instructions and see how they fare.

show 1 reply