Hypothetical Replacement for GEDCOM

[Uncle Buddy]

Here's a quote from an interesting article about GEDCOM. Among other things, the article addresses some genieware provider's questionable practice of encrypting family data. The part I found most interesting is this:

I see one last really good reason for a company to open up its database structure. If they’ve got a really good structure, then maybe others will copy it. If others copy it, then maybe it will become the standard. If it becomes the standard then they are the leaders. Just as FamilySearch was with GEDCOM.

Or even if it doesn’t become the standard, if the database is open, developers can write programs to directly transfer from one database to another without the data loss usually incurred through GEDCOM. This seamless sharing of data with other programs and online trees is something all genealogists want to see.

Followup: Arb pointed out to me on Twitter that MacFamilyTree also uses SQLite and does not encrypt it. Here’s an example of a wonderful way the database was accessed for a Geographical mapping project.

MobileFamilyTree employs exactly the same SQLite database structure as MacFamilyTree, meaning people can use either program with the same database. Now isn’t that a wonderful concept?

There is a common thread here: SQLite.

SQLite is a most incredible phenomenon of the open source software community. Its creator gifted the program to the world with a statement which is a classic in the field of unlicensed, public domain software. His program is one of the most popular and useful programs in existence. The "lite" in its name is unfortunate because it leads some to assume that it's light on functionality. No, actually, it's light on extra effort needed to use it. Unlike other excellent SQL software such as PostgreSQL, you don't need to set up a local server to use it. And when it comes to Python programming, at least if you're using Python in Windows, SQLite comes packaged with the language so you don't even need to install the SQLite library separately. Just install the SQLite tool, the console you type commands into, type `import sqlite3` at the top of any Python module, and you're ready to play with databases, right now. Compared to MySQL, there's little boilerplate. It's very easy to use, and its use of the SQL language is standard. If you already know SQL, you already know most of SQLite.

I said that so I could say this: when GEDCOM was created, no doubt a simple text format was chosen only because, as the brainstorm goes, SQLite didn't exist yet. SQL itself had only become a viable entity a few years earlier.

So while we're playing around with ideas, instead of just eating whatever's served to us, let's pretend we can roll back the clock to 1984 when GEDCOM was created as a sort of intermediary among RDBMS (relational database management systems) and other means of data storage. What if the need for GEDCOM had occurred in a world where SQLite already existed? Do you think we would have seen GEDCOM created as a huge, klunky text file that can take hours to process and then fix thousands of broken data manually?

Here is my hypothesis:

Sooner or later, GEDCOM is going to be replaced by a proper RDBMS database, probably written in SQLite.

Since this means that genieware providers will all have to use the same data storage structure, does that mean there will no longer be a need for independent genieware providers? No. In this hypothetical new world of geniewares that can literally communicate with each other, your GUI and my GUI will and should be very different. And that's all the user cares about. Most genealogists care what flavor of GUI they're using, but few genealogists care what sort of software is used to store their data, as long as it's stored well and not lost. GEDCOM is a big letdown in this area, as everyone knows. It is the very independence of the genieware developers that makes GEDCOM an inadequate solution. GEDCOM is, in fact, a mock relational database. It uses primary keys called "identifiers" and foreign keys called "pointers". It was an attempt to replace a SQLite that did not yet exist.

What if a genieware provider doesn't presently use a SQL database at all? For example, Family Historian apparently stores your data directly into a GEDCOM file as lines of text. My answer to this is that there comes a time in the evolution of an industry where anyone who wants to keep up had better put forth the effort to catch up. If SQLite had been around in 1984 when GEDCOM was created, I'm sure that GEDCOM would not have been created as a text file, and Family Historian would not have been built to store your data in a text file when it came along later.

I watched a video recently where a member of FamilySearch's GEDCOM team said that GEDCOM should not change too quickly. He was talking about some improvements that have been made in GEDCOM 7. Well what if he's wrong? What if we should fix broken things as quickly as possible, even if it means that a lot of genealogists will have to do a lot of work over? Isn't that what we have to do anyway, every time we use GEDCOM? Because every genieware provider stores data differently, a lot of custom tags have to be used to write a GEDCOM file at all. (Actually I suspect that custom tags are overused in lieu of finding creative ways to use the tags that come with the GEDCOM specification.) But it cannot be denied that, for all the good it does, using GEDCOM at all forces the genieware user to do a lot of his work over. Isn't GEDCOM already forcing us to push round pegs into square holes, to say the least? Any file sharing utility is going to force software providers to toe some line. The problem with GEDCOM is that it was invented when the alternatives to it didn't quite exist yet. So are we gonna keep trying to carve granite blocks with wooden mallets and copper chisels, or are we gonna see the light, now that the right tools actually exist, and start using the right tool for the right job?

So what are we waiting for? GEDCOM can and should be replaced by a team-created SQLite data structure that accurately records real people, places, events, and the other elements of genealogy, accurately and realistically, so that GEDCOM's pretense at flexibility--custom tags--can cease to exist. And anyone who doesn't want to participate, well... let them use GEDCOM!

[Uncle Buddy]

Here are some notes I removed from my GEDCOM do list since they turned into a full-blown essay or rant or whatever you want to call it. ("specs" is in regard to the GEDCOM 5.5.1 specs). By the way, if the details annoy you, then just skip to the last paragraph:

Extract conditions needed to validate GEDCOM lines from chapter 2 of specs see p. 19. Start with the number of elements allowed on a line depending on the tag. For example: notes:

Look at exceptions report for Gramps re: NOTE. In fh.ged and gb.ged search `NOTE @` for no results. In gramps.ged this gets 810 results bec Gramps is giving IDs to notes as is correct. Explain why this is necessary and is done by Treebard (so a note doesn't have to be copy/pasted, can be used many times.) Look at specs for NOTE. Both 5.5 and 5.5.1 allow notes to have IDs. Search `Orren R Robertson was born on a farm near Hastings` in all 3 .ged files and both exception reports--Gramps corrected the IDless note imports without mentioning it in the exception report, while FH carried the illegal practice on in their exported .ged...

But is it really illegal? See p. 23 which states that the NOTE line in the HEAD legally uses 3 elements and no ID. And on p. 18 an example of a NOTE line is shown without an identifier. Then on p. 24-26 a note structure is referenced, and this is shown on p. 27. So far it appears to be a requirement that's not required, therefore ambiguous. But the situation is finally clarified when careful examination of the note structure shows that the note itself is bolded but the identifier is not. This means the id is optional. Gramps has shown superior understanding of the situation by fixing un-ID'd notes and exporting them with IDs created by Gramps. But none of this was mentioned in Gramps' exception report because it's not an exception, just a preference.

So the question remains, how to validate NOTEs. Because a NOTE line with both an identifier and note content can have 4 elements. But it seems that Gramps is not doing this, and I agree with this since it creates another condition to search for--four elements in the line--when all the text after the first two elements has already been stuffed into a third element anyway (in my code so far), so there will never be a fourth element. The note content should start on the next line subordinate to the 0-level line. This is the approach Treebard will take: if notes are imported without IDs, then IDs will be added, with the note content starting on a subordinate line in Treebard GEDCOM exported. And all imported lines with more than three elements should be rejected and added to the exceptions log, as far as I can tell at this point.

It looks like my confusion with the 5.5.1 specs in regard to the NOTE tag was shared by others, because if you look at the specs for GEDCOM 7.0.7, the option of using an identifier with a NOTE tag has been removed, and instead there is an additional shared note tag `SNOTE` which does have an identifier and can be referenced from a variety of places. Making the note re-usable.

Which means to me that the current effort to coax GEDCOM into the 20th century is still what it always was: an effort born in 1984 to replace SQLite before SQLite was born. By which I mean that GEDCOM was created in 1984 when SQL was brand new and SQLite didn't exist. Now, instead of trying harder and harder to make GEDCOM work like a proper RDBMS, we should admit that SQLite took over the world (for good reason) while GEDCOM languished without an update (also for a good reason: a better solution had already come along). The punch line is that the valiant attempt to make GEDCOM act like a proper RDBMS is as wrong-headed as it is valiant, because SQLite & friends do in fact exist, are stable and entrenched into the way everything else is done... so why doesn't the genieware development community get with the 21st century and admit that GEDCOM is no longer a valid tool, since a standard for sharing data does now exist: the simple-to-use, serverless SQLite database.

[Uncle Buddy]

Why is GEDCOM still being used, in spite of the extra work needed to keep it alive, not to mention the work needed to get it up to speed at the task of describing the world accurately? Why do we insist on redescribing the world with a text file after already having done so with whatever sort of database is peculiar to each different genieware provider?

The sort of genealogist who wants to be on standards committees for creating and improving universal tools like a file sharing utility such as GEDCOM, is the sort of person who would tend to like GEDCOM and keep it in the arsenal as top dog in file sharing. Why? Because it's humanly readable so there will be more people able and willing to come to the table and discuss the problems that need to be solved in order for committee-types to enjoy themselves. Why? Because they don't want to be a committee of one with nobody to interact with, no audience, no voice, and no influence.

Not everyone who wants to help create and modify standards wants to learn SQL, but every single one of these committee-type genealogists can look at a GEDCOM file and figure out how it works and develop opinions about how to improve it. (Which just amounts to fiddling with it for fun because its reason for existing has silently been canceled by SQL.) The text-file solution is completely accessible to anyone who understands the needs of genealogy, so committee types keep it around because it undershores the very existence of its supporting committees. Do we need standards committees? No, we need a fun hobby, and that's what genealogy is for 99% of the people who try to find a decent GUI to use, and kinda can't find one.

The solution is for you and I to write our own genieware and have our own sharing utility which is a SQL database, and let the committee types catch up when they're tired of endlessly fiddling with GEDCOM. If you're a committee type, don't take this the wrong way. My definition of a committee-type person (which I just invented on the spot) is someone who gets volunteered for something he doesn't have time for and due to the added stress of yet another superfluous task, ends up taking it out on the world by inventing a bunch of rules for other people to follow.

So don't be mad at me, I'm trying to sympathize with what it must be like to get a group of people to agree about anything. In the meantime, people like me still have to use GEDCOM because we are committees of one. The person most likely to listen to a committee of one is an anti-committee type, and these folks are by definition never going to outvote those who auto-cluster in the center and vote in groups based on groupthink. Sure they listened to Galileo... after he was dead.

Next time you're waiting hours for a GEDCOM file to process, just kill some time by googling this:

which is faster for a computer to read text or binary