Беды с контекстом письма

Вот хорошая вырезка, демонстрирующая главные проблемы которые вижу я в ЛЛМ:

I’ve drafted, redrafted, scrapped, rewritten, and cast my literature review into hell so many times that I’m painfully familiar with the topic (broadly: teacher-writers and digital writing, which now includes GenAI), so I figured it would make for an interesting comparison with Deep Research’s attempt. I mentioned earlier that the application cannot access paywalled articles, so I’m not judging it here on its ability to find the same breadth of sources I can access through my university library – that would be an unfair comparison.

But I am interested in what it can do with the sources it has access to. Unfortunately, the finished report made some classic lit review errors which you’d expect to be addressed at an undergraduate level, or maybe in a Masters. The entire (16,000 word!) review was filled with description and summary, but lacked any analysis. Passages like this:

The use of verbatim quotes with no further explanation is a no-no, as is the fairly banal closing statement “as Bandura’s theory of self-efficacy would predict”. It mentions “quantitative data” with no exploration of why that’s important (for context, one of the biggest criticisms of research in this field is the reliance on anecdotal and qualitative data). There is no follow through, and no discussion of why this New Zealand research is important in the global context, or at all, really.

It did find some of the seminal (open access) papers from the field which I had used in my own literature review, but it lacks the capacity to articulate why they’re so important. Even at a surface level, you might assume that an application of this capacity (and cost) could make some judgements of the relevance of a piece of research based on its citations, appearance in other studies, or a similar metric. All we get here is a list of papers, with the ever-present ChatGPT bullet points:

There has been some back and forth from defenders of the “PhD in your Pocket” that Deep Research doesn’t replace a PhD, but that it can replace some of the skills of a PhD, like Mollick’s substack above suggests.

Maybe I’m using it wrong, and perhaps this will continue to improve over time (another classic defence of the AI boosters – you’re prompt is bad! Just wait six months!), but I’m just not seeing it yet.

<...>

The only conclusion I could arrive at is that it is an application for businesses and individuals whose job it is to produce lengthy, seemingly accurate reports that no one will actually read. Anyone whose role includes the kind of research destined to end up in a PowerPoint. It is designed to produce the appearance of research, without any actual research happening along the way.