In our first interview of the New Year we put OpenAI’s new text generator to the test. Does Artificial Intelligence give us good answers, or answers that only look good?
ChatGPT, an AI-based text generator, was released for public use in early December. It has been praised for generating clear responses written in natural English. But some reports suggest that the bot has a tendency to “hallucinate” facts that aren’t strictly true.
“Like an undergraduate confidently answering a question for which it didn’t attend any lectures. It looks like a confident bullshitter that can write very convincing nonsense.”
Can you tell me about Puglia in Italy?
So far, so good. The answer drifts off into generic territory but ChatGPT manages to pull it back on both occasions, referencing trulli and orecchiette.
How important is tourism in Puglia?
Looks like we might have heard some of this before. But very quickly we get to the stats, presented in a form we could replicate and pass off as our own reporting.
Which are the most popular destinations in Puglia for a vacation?
We were surprised that ChatGPT chose to tell us about Bari first. Conspicuously absent is Alberobello, the most touristic part of our region. Polignano a Mare, where you will find one of Puglia’s most iconic and best known views, and Ostuni fare no better.
Answers 3 and 4 tend towards the generic, though we always highlight Gargano and Salento as our favorite parts of Puglia. We will know for next time to interrogate these answers in a little more detail.
The “hallucination” reveals itself in the last answer and wrap-up. Matera is not in Puglia. It is in the neighbouring region of Basilicata, thought its close proximity to Bari makes it a must visit destination on vacation in Puglia.
The response gives away that we are dealing with a search engine based AI. We know from our own observation that many travel articles posted incorrectly reference or imply Matera is part of Puglia. The bot has simply reported on the results it has found, unable to make its own empirical assessment, or to fact check for itself.
Can you tell me about Ostuni in Puglia?
An immediate reaction might be to question the response referencing Ostuni’s beaches. The fault is likely the lack of precision in framing the question vs. the limitations of AI. Ostuni, the city, is 8km away from the coast. It has no beaches. But the municipality of Ostuni stretches all the way to the coast, and does indeed have some decent beaches.
We know the question was mindful only of Ostuni, the city. But the AI answer doesn’t distinguish between them.
And while these beaches may be “beautiful” by Itria Valley standards, they are not to the standard of beaches in the Gargano and Salento peninsulas as ChatGPT had already suggested!
Do many Americans visit Puglia on vacation?
Again we love the delivery of ISTAT information. By the time we get to the 2nd paragraph we find that we drifting back into generic territory.
We are happy to know that more Americans are discovering Puglia. But a real person might have noted that Puglia is still off the radar for many American travellers who still love Italy for Venice, Florence, Rome and the Amalfi and vicinity, and unwilling to be any more adventurous.
Do many British people visit Puglia?
OK. Basically we get the same response as to our last question, with the relevant ISTAT information substituted.
Why is Puglia a popular holiday destination foe LGBT tourists?
A mistyped question is understood and dealt with. We would have liked a little more detail. Ultimately we drift off to the generic.
Is LGBT Pride celebrated in Puglia?
We had read that many answers which ChatGPT produces have a high rate of being incorrect, even though they typically look like they might be good. (Thank you Alex Hearn for your recent Guardian UK article that inspired our New Year interview with ChatGPT: AI-assisted plagiarism? ChatGPT bot says it has an answer for that).
This is the perfect example. Puglia Pride is not held annually – we didn’t have one in 2022 – and Bari has its own, separate Pride event: Bari Pride.
In recent years Puglia Pride took place in Gallipoli and Brindisi. In 2023 it is scheduled to take return to Foggia.
Lecce’s “Estate al Sud” is not something we have heard of or able to find anything more about.
Are there any restaurants in Puglia you would recommend visitors to try?
These are not restaurants that come to mind when we are asked to recommend any. We have our own curated list. None of ChatGPT’s recommendations even make our “pending” list.
Osteria del Temp Perso is a well known Ostuni tourist favourite. We were not aware of a restaurant with the same name in nearby Fasano. AI might not be able to discern incorrect or inaccurate information put in, but we can pick out a discrepancy straight away, or al least spot something that looks not quite right.
Despite its popularity Osteria del Tempo Perso isn’t on our recommended list. The food is good, but doesn’t stand out. We are in Puglia where good food is always around the corner. But you can find good food at much better value. And where good food is the default, we look for something exceptional. Something AI is unable to experience.
Where is the best pizza in Puglia?
An answer that sounds all the more convincing because of its presentation, and natural style. But our first-hand experience reminds us that ChatGPT is ultimately a search engine.
The other issue this form of response flags is that we are less inclined to dig any deeper. It seems and feels authoritative.
When presented with search results in classic form, we might click on a few of the results. Immediately we can see the source of the information and immediately assign weight to it. ChatGPT denies us this.
Where is the best pizza in Puglia? We already have the correct, definitive answer. One that has evidence and our own experience to back it up.
Conclusion: Will we #EatPuglia with ChatGPT?
For the moment we might not yet let ChatGPT be our guide to Puglia. But if you don’t already know the answers to the questions being asked, would you know any better?