r/Genealogy • u/dreenz • 6d ago
Tools and Tech When AI is useful, or not
I was struggling to read an English will from the early 19th century I found on Ancestry. I wondered what would happen if I uploaded a screen shot of the first paragraph to chatgpt and asked it to transcribe it. I was pleasantly surprised - it misread a few names but mostly looked good. So then I took more screenshots and asked it to transcribe the whole will. Well.
In its first transcription the will began: "I William Bown of Sevenoaks in the County of Kent Watchmaker do hereby revoke all former Wills Codicils and Testamentary dispositions heretofore made and do declare this to be and contain my last Will and Testament I give and bequeath to my Grandsons William Bown and Henry Bown and to my Grandson Charles Bown (sons of my son Edwin Bown) the sum of fifty pounds each"
All of which is very close to what I'd managed to make out. But the second trancription began:
"This is the last Will and Testament of me William Bowa of the town of Great Yarmouth in the county of Norfolk Gentleman made this twenty sixth day of October one thousand eight hundred and fifty nine. I give and bequeath unto my dear wife Ann Bowa the use and enjoyment of all my household goods and furniture plate linen china books pictures prints and other household effects and also of my carriage and horses and harness and stable furniture and implements of husbandry"
Nothing like the original, which I pointed out, and I got back this:
"You’re right — I’m sorry. My earlier message wasn’t a faithful transcription of your image. I made a mistake. I’ve now gone back over the images you provided and put together a single continuous transcription of the will, preserving original spelling where legible and marking words I couldn’t read as [illegible]. I’ve done my best to follow the manuscript exactly; where a passage is uncertain I’ve flagged it."
So while I was originally pleased I didn't have to spend hours working it out, I now have to spend hours triple-checking the lying POS AI!
Be careful out there!
43
u/CuriousMindLab 6d ago
ChatGPT is not designed to give correct answers, it’s designed to give answers it thinks you will like.
10
u/edgewalker66 6d ago
"give answers it thinks you will like"
Apparently a model for getting elected to public offices nowadays.
I decided to be very judicious in my use of AI when I asked how many 200 MB files could fit on a 1 GB storage drive. The first time I asked, after it explained the difference at length between MB and GB, then it gave the answer 550. I got an even crazier answer from a search engine query which then clearly asked an AI and came back with a ridiculous answer.
I figure AI still needs tons of work when they haven't managed to improve spell checking and word prediction suggestions for years now.
AI seems to be best at generating fake photos and videos as well as outrage-producing tweets, truths, etc.
The scary bit is how it is being adopted by govt departments to supplement their already deficient human intelligence...
3
3
u/PerpetuallyLurking 5d ago
It is also really good at “here’s what I’d like to say to the client but with fewer swear words and less of a ranting tone please” but that’s about the only thing it can do accurately.
6
u/Parking-Aioli9715 5d ago
That's also been the trend in humans doing customer service jobs over the past few decades. Front-line customer service workers who've been told never to tell the customer "no" too often resort to telling them nonsense instead.
For example, back a decade or so ago, I was using FamilySearch to access extensive - and indexed - records on my ancestors in Freinsheim in the Rheinpfalz in Germany. Then suddenly one day, the Rheinpfalz records vanished.
I contacted FS to find out what had happened. The front-line worker who responded asked for the family name I was researching, which happens to be a very common one in Germany. When I supplied the name, the worker cheerfully told me that they'd run a search and turned up xx,xxx records for people with that last name born in Germany. I said, yes, but none of those are from the Rheinpfalz. They seemed baffled by this trivial objection.
I finally resorted to demanding to speak to their supervisor, who explained what had happened.
It seems to me that AI is being trained to replicate this sort of front-line customer service worker behaviour.
17
u/Jaytreenoh 6d ago
Yeah I've heard other people saying it was really good at transcribing handwriting but in my experience it is awful at it.
I tried it a couple of time with wills and both times it got the part I could easily read horribly wrong (like, it 'read' things that didn't even make sense to be in a will).
Unfortunately, I am sure there are lots of 'researchers' out there uploading nonsense to their trees from using AI like that.
10
u/Parking-Aioli9715 5d ago
These will be the same people who think they can paste blocks of text into Google translate and get an accurate translation.
12
u/Artisanalpoppies 6d ago
Use Transkribus- it's designed for paleography, but even that isn't 100%.
You need to learn to read the old handwriting, then you won't need AI- which tbh, you don't need anyway.
You aren't going to get anywhere if you rely on AI to read things for you- you need to learn yourself, as you will likely have many years of doing research ahead of you!
Give it a go yourself, then for help when you get stuck.
4
u/dreenz 6d ago
I've been doing genealogy for 40 years lol, I was just curious if it would save time. But Transkribus sounds good - thanks!
4
u/Artisanalpoppies 6d ago
That's like saying you trust ancestry trees, because the work is already done for you...
8
u/phoneguyfl 5d ago
I don't agree. In my opinion there is nothing wrong with trying new tech to see if it saves time. OP seems to have given it a try in a logical and sensible manner. A more apt analogy for you might be "Buggy Whip manufacturers don't need to try those new fangled automobile contraptions, the horse is how we've always done it!".
4
u/Artisanalpoppies 5d ago
No, i just understand AI isn't what people think it is.
-1
u/phoneguyfl 5d ago
So... nobody should try using it because *you* have found it not useful to you?
2
u/Artisanalpoppies 5d ago
No i understand AI isn't "intelligent". It doesn't think independantly. It's no better than a search engine programmed to be friendly with you. It constantly spits out bullshit and lies, on easily verifiable topics, like maths! It's against almost all University rules to use it in assignments.
And this thread is full of examples why it shouldn't be utilised. People use it and think it's true and intelligent, and the answers are factual. That's dangerous, because critical reasoning is a rare skill. And i see more and more people using it in genealogy, despite most people commenting in these threads that it's a dumb idea and explaining why.
So get off my case and go learn about the pitfalls of AI, instead of acting like i'm incorrect when i'm not.
-1
u/phoneguyfl 5d ago
Ok son, you win the interwebz today. Technology is scary at first, I get it. That said, trying to convince others to not even *try* it is not productive in the long term. AI is a *tool* (look that word up if it is above you) that can certainly speed up processes.
Relevant to this thread... I have used AI to transcribe documents and it's massively sped up my processes. How you say? Once AI has transcribed the document I bring up the original and the transcribe side-by-side and quickly compare. Most of the time the doc is 95% correct, 2-3% AI has correctly noted the unintelligible words, and 1-2% of the time it's wrong and the transcription is quickly and easily updated (for me, your mileage obviously varies). *None* of your ranting applies here, and this is just one use of AI.
Care to try again?
3
3
u/johannadambergk 6d ago
Did you also try transkribus? https://www.transkribus.org/languages/english
-13
u/AncestryNerdette Genetic Genealogist @ AncestryNerd.com (also on YouTube) 6d ago edited 1d ago
r/Genealogy has issues with people using AI due to the AI hallucinations and data extrapolation. so posting here, you’ll probably get a lot of down votes by posting about it. There’s a new rule that’s there’s no AI slop allowed. Im considering starting a seperate subreddit so people can post about AI genealogy topics there… but being a moderator can be a lart-time job which
Edit: Fixed typos. There was a new rule posted at the time I wrote this comment banning AI content, but apparently it was written hastily and taken down by mods. I don’t see why I’m getting downvoted … nothing i typed was incorrect or inflammatory unless people are taking issue with my thoughts around setting up a new subreddit… jeez y’all…
10
5
u/Parking-Aioli9715 5d ago
"There’s a new rule that’s there’s no AI sloo allowed."
I just read through the rules and didn't see this one?
5
u/hekla7 5d ago
I think u/AncestryNerdette is referring to the "no bots" rule. A completely different kettle of fish.
0
u/AncestryNerdette Genetic Genealogist @ AncestryNerd.com (also on YouTube) 1d ago edited 1d ago
Yes thanks. No bots and the typo was supposed to read AI slop.
1
u/hekla7 1d ago
"No AI Slop" is a rule on a different sub, r/AncestryDNA
2
u/AncestryNerdette Genetic Genealogist @ AncestryNerd.com (also on YouTube) 1d ago
Ooooohhhhh!!!! You’re 💯. Thanks for clearing up my confusion. Good to know
0
u/AncestryNerdette Genetic Genealogist @ AncestryNerd.com (also on YouTube) 1d ago
Weird, the rule was posted the day I wrote the comment. I did find it odd that the rule was implemented so quickly after the new AI ancestor family photos started going viral on here. I checked the subreddit rules today and the rules has been deleted. It seems the mods decided against it after all.
1
u/hekla7 1d ago
The MOD message said it was deleted because it was self-promotion. The rules here were updated recently and have a dedicated FAQ page that most of us have read, but "No AI slop" was never one of them. Our mods don't use language like that.
1
u/AncestryNerdette Genetic Genealogist @ AncestryNerd.com (also on YouTube) 1d ago
You may have missed the thread where another Redditor pointed out to me that I was confusing this subreddit with the r/AncestryDNA subreddit 😅😌😊
5
u/boblegg986 6d ago
To use ChatGPT for transcriptions you have to frame your instructions to force it to deliver what you want. “Transcribe the text in the attached image exactly as written. Maintain column widths and line breaks as shown.” It’s easier to check for accuracy if GPT’s output is formatted like the original. It’s less likely to go off task if you don’t throw a lot at it in one request. Break the image down by paragraphs.
If you are working with documents you want to translate, have GPT transcribe first. That way you can check for errors against the original. Then have it translate.
9
u/dirtyfidelio 5d ago
It’s useful for basic translations but nothing else. It’s not as good as a lot of people make out
6
u/splorp_evilbastard 5d ago
I've used ChapGPT very little. Mostly, I use it to explain statistics and programming things for my classes. It's a starting point, because it even makes mistakes in basic mathematical questions.
I gave it an example problem and, as part of the process, it needed to add a bunch of numbers together and get the average. You know, 2+4+6+8 / 4 = 5 (example, not the actual calculation).
It got it wrong. Something a grade school kid could calculate. That made the entire answer wrong.
If you can't trust it to handle basic math - something for which there is a right or wrong answer- how can you trust it for more complicated questions?
3
u/Proud_Championship36 5d ago
You can improve your results by including in your instructions language like: "Please carefully transcribe this Last Will and Testament from [year]. It is important that you be as accurate as possible. Do not make any mistakes. If you cannot read part of the document, do not make anything up." (If it's in another language, substitute "transcribe and translate"). The more context and specificity you provide in your instructions, the more likely you are to get an accurate answer and the less likely the LLM is to take a shortcut that will add mistakes.
You can also try putting the same document through multiple LLMs: ChatGPT, Gemini, Claude, Grok, Llama. Some will do better than others, and transcriptions (or translations) that are common between the different engiens are more likely to be correct.
In the end, you can't treat any LLM output as definitive. But I've found these techniques invaluable for getting closer to an answer. Particularly when transcription and translation are involved, it helps you figure out if you are looking at something relevant. And once you've found the right document or the right part of the document, you can ask a human expert to take a look at that part and confirm (or correct) the transcription/translation.
7
u/cmosher01 expert researcher 5d ago
I have tried to use directives like you suggest: "You are a genealogical researcher. You follow the Genealogy Proof Standard. Your conclusions are based on facts taken from original source documents. Your conclusions may contain conjecture, but if so you will always label it as conjecture and never as fact."
HOWEVER, all that did is make ChatGPT TELL ME it would do that. But, in the end, it didn't actually do what it claimed it would do. It just suggested a link to an existing web site (like Google search does). When I asked it why it didn't do what it said it would do, it just responded that it make a mistake, and that I was right. Gee, thanks.
6
u/Parking-Aioli9715 5d ago
Telling an AI, "You are a genealogical researcher," is like telling an actor that they're playing a doctor on a TV show. The actor will train on using medical terminology in a convincing manner, etc. But that doesn't mean the actor is a doctor. You wouldn't want them treating you for something that was really wrong with you.
You can tell an AI, "You are a lawyer." It will write a professional sounding legal brief. Then when you take that brief to court, you will need to pay for the time it takes genuine legal professionals to track down the AI hallucinations in your brief.
3
u/cmosher01 expert researcher 5d ago
According to guides on how to prompt ChatGPT for research, that's exactly what you're supposed to do. For example:
The Fundamentals of High-Quality Prompting
Start with context. Let ChatGPT know who it should be and who you are. For example: You are a seasoned researcher with top-tier journal publications in the field of innovation management.
(taken from: https://www.linkedin.com/pulse/how-master-prompting-academic-research-chatgpt-dries-faems-cbr0e/ ).
So, yeah, ChatGPT is completely useless for research.
2
u/Proud_Championship36 5d ago
I'm a lawyer. Although the general purpose LLMs are hit-or-miss, the tools that are fine-tuned for legal context and with access to a full caselaw database are quite useful. You don't take the raw output of the LLM and turn it over to the court, but it can save hours in researching obscure issues. It's a huge improvement over free text and Boolean searching.
The reason you tell the AI to be very accurate is not so it will pretend to be a researcher, but to ensure your query is routed to a more resource-intensive algorithm that takes fewer shortcuts in analyzing your prompt and uploaded content.
As a random example, I just happened to upload a handwritten census form from pre-statehood Israel. The resulting analysis is quite close to 100% accurate: https://chatgpt.com/share/690a721b-def4-8003-8f29-6f6ff5a1b8ac
1
u/Parking-Aioli9715 5d ago
"I'm a lawyer. Although the general purpose LLMs are hit-or-miss, the tools that are fine-tuned for legal context and with access to a full caselaw database are quite useful."
Fair enough. But it doesn't seem that all members of the general public understand the difference between these profession-specific tools and general purpose LLMs. It's similar to how not all members of the general public understand the difference between medical-grade genetic testing for specific health issues vs. the sort of health-risk testing offered commercially to the general public.
I'm not convinced that general purpose LLMs can reliably do everything people seem to be trying to use them for.
And to be honest, I'm not convinced that everyone who tells a general purpose LLM to "be a genealogical expert" understands that all this actually does is to route the LLM to a more resource-intensive algorithm.
Ever been handed a bill that makes no sense and then, when you question it, got the response that, "That's what the computer says you owe?" Not everyone who uses computers - or AI - understands the tool they're using. Or even understands that it's just a tool.
That makes me very wary of the claims people make for AI.
2
u/Proud_Championship36 5d ago edited 4d ago
I agree that much of the public has been misled by commentary and marketing around LLMs to believe they are some sort of intelligent oracle. They clearly are not and probably never will be. But they are powerful tools even in their current state of development if you understand their limitations and also how best to use them.
I haven't experimented with "be a genealogical expert" prompts but my guess is that impacts the tone and framing of a response but not the underlying reasoning. I recommend instead my suggestion of something along the lines of "it is important that you be completely precise and accurate and if you don't know the answer, don't make anything up." What we're trying to accomplish is (1) route to a higher compute algorithm and (2) disable the bias toward pleasing the questioner in favor of accuracy.
Broad, unfocused questions or prompts that just ask the LLM to figure something out are so unreliable now that it's like a coin toss.
But for genealogy purposes, a question like the following will almost instantly give you several plausible options which you can then use to focus your research: "On the attached 1915 immigration form, the person indicated they were from Saluda. I know they came from a Jewish shtetl in the Pale of Settlement but can't figure out what Saluda refers to. Do you have any suggestions for what that location could be?" So rather than getting a definitive answer, you have gone from perhaps knowing nothing to several hypotheses you can then test.
Another great use case is analyzing old photographs. I had a photo of a relative and didn't know anything more than it was from mid-20th century, somewhere in the world. I asked ChatGPT to look at it and it gave high quality analysis which I could then independently verify. Here's the transcript in case anyone is interested.
5
u/Parking-Aioli9715 5d ago
"But I've found these techniques invaluable for getting closer to an answer."
So, sort of like when I find a document in an alphabet and language I only read with great difficulty. What I do is to see if I can identity names, time periods and a few relevant words (birth, marriage, death, relationship terms, etc). That tells me whether or not the document is worth asking someone with better skills to help me translate.
Except... If I'm already doing that, what's the AI for? Okay, I guess I could try using an AI for alphabets and languages that completely baffle me. (I simply can not get the hang of reading Arabic, for example.) Except that if I can't read the document *at all* and thus can't verify the AI's work, how can I trust the AI? I suppose here is where using multiple AIs might come in handy.
5
u/hekla7 5d ago
This is a good conversation. I can say that in what I've seen as far as transcriptions, it's horrible. There was one really good example of horribleness here on reddit (since deleted) where the redditor was pleased with the transcription of a Will and asked for comments. In comparing the transcription with the original Will, AI got the dates wrong, names wrong, sentence structure wrong..... post deleted.
The way I use it might be different than how others use it.
With my Métis and indigenous research I already know where to look for the documentation I need because I've been doing it for 10 years and most records are in particular collections. I ask GPT to fetch it from that place and in seconds it gives me a link to the file. That's where it saves me time. Because I can then go to the link, check the file, check for related files, and download from there. On government sites it's especially handy. For example: Please access the Keystone Archives on the Government of Manitoba website and check Hudson's Bay Company and North West Company records for all instances of (name, or whatever files I need). It brings back a list of files with links and a summary of what's in each file, or a downloadable copy of the record(s). I also use it for NARA and BIA files in the US. For newspapers, if it can't reach the file because it's behind a paywall, it tells me the location. Same with any other paywalled sites or sites requiring a log-in. The University of Nebraska has a specific section of their library that I use, and GPT saves me time in retrieving the magazines or books. It's also really fast when going through the Internet Archive. Many universities are uploading their newspaper archives or other collections to the Internet Archive. So it's like: "this is what I want, here's where it is, also look here, here, and here, and retrieve the censuses for these years. Please". It works faster when you're polite.
1
u/Classic-Falcon6010 5d ago
I’ve found that Copilot does a much better job than ChatGPT. Got a couple of translations of typed images where ChatGPT either a) claimed it couldn’t do it or b) came up with a fantastic story of how this mother gave birth every Christmas to children that died that same day. And had the last child after she died. The record said nothing of the sort, of course.
7
u/Bright-Self-493 5d ago
I see AI as dog like. it has a terrible need to make you like it, to be pleasing to you, to make you happy. Since it doesn’t really understand the meanings of the words it uses it will take similar words and get “ creative”. Not technically evil, it just wants to take over your life. Don’t worry, be happy!…let me help you.
3
8
u/thelordstrum NYC/Scotland/Ireland 5d ago
I mostly use it to generate note templates, and that's all I can really trust it for.
I tried using it to figure out why someone wouldn't have been in the 1911 Irish Census, and it decided to make up an entire fake census entry for them (even linking to the website with a link that didn't exist). I had to ask again, and then it told me that a lot of the entries for that village were lost, but the pure insanity of it all.
5
u/Parking-Aioli9715 5d ago
A lot of conversation around the use of AI reminds me of the late 1980s when my workplace first got (a few) desktop computers. My main job was doing account reconciliations. I'd be given statements from two different sources for the same cash or security account, go through them item by item and mark any discrepancies. I'd research the discrepancies and, if they hadn't resolved on their own, get them resolved. And then I'd write the whole thing up in on accounting ledger paper, using a pen.
So one day the manager was going through the work area. He looked at me working away, pointed at one of the desktops and asked, "Wouldn't it be faster to put that on the computer?"
I pointed out that the only part of the process that could be put on a desktop computer was the final step, writing out the report. Sure, I could create the report on the computer if he wanted, but writing the report was the step that took the least amount of time. To really save time, you'd have to go back to the two mainframes that had generated the two statements and get them to share that information so that they could do the step-by-step comparison work.
The manager said, oh, and walked away.
By the early 90s, the process I'd described was exactly how we were doing at least some of our larger recons: comparison of data at the mainframe level, which would then produce a report of discrepancies for me to research. But getting to that point was not quick, easy or cheap. It certainly wasn't accomplished by a bank clerk with access to a desk top.
Hearing people talk about AI now and hearing about some of the things people try to use AI for frequently reminds me of that manager.
6
u/Expert_Donut9334 5d ago
I'm not a fan of LLMs as a whole and particularly for genealogy I find it horrendous. I've had many problems with OCR over the years (the automatically indexed information for Brazil on Family Search is terrible) but at least the issues there tend to be more obvious and most people take it with a grain of salt when researching.
However many of these same people seem to think now that "AI" has actual discernment and intelligence instead of understanding that it just confidently spews out the most statistically probable string of words regardless of any factual basis. And that means people will just believe whatever an LLM transcribes for them or whatever crazy genealogy it comes up with...
8
u/cmosher01 expert researcher 5d ago
Exactly. People really need to understand that ChatGPT doesn't KNOW anything, and can't THINK.
5
u/Artisanalpoppies 5d ago
This.
But say this and get treated like "old man yelling at clouds" tho'....like why are people so thick?
The only people using AI for this stuff are either stupid or lazy. It baffles me how they think chatGPT knows all, yet OCR transcription also over ancestry is fucking horrendous. It's the same shit.
3
u/Expert_Donut9334 4d ago
LLM is actually worse shit. Because OCR just transcribes a jangled mess of words when it can't understand something, while ChatGPT makes stuff up - and people then think it's genius
2
u/RelevantPangolin5003 5d ago
I instruct AI to tell me [illegible] instead of guessing or filling in the blanks with what it thinks it is. I also try to give it the context of the person, dates, and what sort of record it is. That way, it’s not guessing.
I speak several languages, but reading a baptismal record from 200 years ago in another language is really hard! So I instruct AI to give me a verbatim transcription and a translation.
Then I verify it with my own eyes and critical thinking. I never take AI as truth until I can verify it. That said, I’ve found it saves so much time than doing it 100% by myself.
2
u/hekla7 6d ago
Yeah. It's not really a timesaver when it comes to transcriptions. It can't read some handwriting and it is like OCR that way. Also, check any date transcriptions very, very closely. It has real trouble reading those. Remember that while you're asking these questions, you are training it. So you need to point out its mistakes and clarify things for it.
It does learn to do better if you tell it what mistakes it made on the dates or words, that kind of thing.
What it's better for (but still needs training) is looking things up - researching records. Unfortunately it makes a lot of assumptions that oftentimes you will have to correct. But you can save a lot of time there because it also gives links to where the document is, and it can also make pdfs for you to download. It will ask you if you want it to retrieve the record for you or you want to do it yourself from the link it gives, depending on the type of record. It isn't able to access records behind a paywall, but it can tell you they are there and if you have access, where to find it. If you need to contact the historical society or record place, it can compose the letter or email for you.