Families Table--starting over again?

[Uncle Buddy]

It's too late in the day to decide for sure but it looks like tomorrow will be another fresh start on the families table. I took a long break from it and when I got back to it, I found it a brittle masterpiece of bandaids and patches that took shape as a sort of tentative fix layered on top of a trial run on top of a groping in the dark. The result is a nice-looking families table that almost works, but the code looks like a Picasso; fixing one thing breaks another. When things get to this point, a restructuring is generally the most efficient strategy. Anything else is just gonna be another layer of bandaids on something that wasn't planned very well from the start.

I believe this is also how the Great American Novel gets written. There might be some writers who just pop out finished masterpieces the first time, but I'd guess that most writers are just learning the territory the first and second times through. I can't tell you how many times I've rewritten this feature. I don't remember.

The general design will be kept. Treebard's families table is a scrollable window inside the current person tab, above the events table, occupying about 2/3 of the width of the screen. At the top of the families table is a bordered area for the parents and alt parents of the current person. There are two permanent inputs for biological mother and father. Under that, if the user creates an adoption, fosterage, or guardianship event, two more inputs appear for alt parents. Generic labels such as "foster parent" can be changed to "foster mother" or "foster father", for example, by double clicking the label to open a tiny dialog.

Under that is the progenies area where every one of the current person's partners and children is displayed at the same time. A vertical scrollbar will appear if it's needed. I believe that displaying all the person's families at once is unique to Treebard. Most apps require the user to click away from one family to view another, but Treebard's goal is to show all at once the whole story of one person's life without having to visually leave one place and go to another where you will have to try to remember what you just saw. Or click back and forth between them, for example to compare dates. The family table will be sorted by date. Partners are added to the table based on both marital events such as wedding, marriage, divorce, etc., and on child events such as a birth or adoption.

Partners will be labeled the way that the user enters them, not automatically as "spouse", "husband", or "wife". Treebard never assumes moral or religious traditions, but the user is free to do so; he can call anyone a spouse if he wants. I ran into a situation where a couple had a daughter and an adopted son. So on the man's current person tab, his wife appears in his families table labeled "Mother or guardian of children" with the two children listed under that. In the events table below, pointing at the guardianship event tells you which child they were guardians of. Pointing at the offspring event tells you which child they were the parents of. Just while writing this, I realized the label will be even more complicated if I remember to give the couple a marital event. Something like: "Legal guardian or mother of children (wife)".

All this is not as easy as it sounds, but historical accuracy demands it. If Treebard is going to be an example of a family-relationship structure that reflects the real world, then that is the goal, not some intermediary goal along the way. Treebard has to prove that the difficulty of doing it right is not rocket science. If I can do it, any real programmer and even a lot of amateurs can do it better. But the data structure behind this is simple. Not as simple as existing genieware, but simple nonetheless. So if a genieware creator doesn't want to try and do so much in one families table, this same data structure could be used behind any number of different GUI designs. But Treebard is where the action is right now, so:

In the progenies area, each partner has a heading where his/her kin type will be labeled, e.g. "partner", "spouse", "father of children", followed by their name.

Separating the partners from each other, there's an indented list of that partner's children with the current person. Each child's name, gender, birth date and death date are listed in a table.

Under the progenies area, an input field allows the user to enter new children for the selected partner. I haven't even started this important feature. I just realized that the only reason I haven't started the Add Child feature is that it's at the bottom. That's not a good reason. I think I'll do that part first this time, in case it's not easy.

All the person inputs in the families table are autofills, so the user can start typing and tab out when the right person fills in, or input a new name and a plus sign to add a new person, or a number sign followed by the person's ID, or any name stored for the person, such as a nickname or a membership number down at the Elks Lodge. If the name is spelled the same as an existing name in the database, a dialog will open so the user can select the right person by that name or create another one. Any of the person inputs that contain a name can be double-clicked to change the current person to the clicked person. Pointing at any of the names will open a small tooltip which will display the person ID and name type of the pointed-at name.

None of this is theoretical, it all works. Except the feature I mentioned which I haven't started. The problem is that it's so complex, and has been patched together in a series of experimental dashes for the finish line, that the result is a maze of code that can't properly be finished. It's gotten to the point where fixing one small problem creates another big or small problem. Staring at the code for hours doesn't encourage it to fix itself. Life doesn't have that many hours in it. So to save time and frustration, I will keep the GUI as is and rewrite all the code that makes it work. As usual, I'll trust that I've learned something since I started the families table, and literally delete all the logic code and rewrite it from scratch. I've already rewritten some features more than once, and tossed out other features that were redundant or supefluous. Now that I know what my goals are, it will still be hard to get this much functionality into one small table, but it won't be impossible.

What never works for me is using broken or fragile code as the model for new code. The key to my strategy, which I've used over and over, is to not even look at the old code except for very specific segments that can be plugged in to save time. Just delete it and start over, trusting that I've gained enough experience to do a lot better this time. This sounds more time-consuming, but it's not. I don't have to bang my head against a bunch of patched-together code, trying to figure out how it worked, or which part was broken, or which part was conflicting with what other part. Or how many lists & dictionaries I've created to hold the same data.

One reason this families table has gotten so discombobulated is that in order to even get this far with it, I've had to take time out to restructure older, big functionalities. Since the time, long ago, when I finally bagan the families table, I've totally rewritten the way that new color schemes are communicated to the app. Treebard is built around the user's ability to make his work easy to read and easy to look at. Fonts can be changed, and I had to do another full rewrite on that functionality. I made a huge change in the database structure which makes everything easier. So this time, when I start writing the families table logic, I'll be standing on much firmer ground and have a sort of mental map of how things should be.

The screenshots show a big complex family table scrolled to its two extreme positions. I think all this trouble is well worth it so the user can see a simple picture of a person's life with the flick of a scrollbar or mousewheel instead of clicking back and forth among several families displayed in different places. (All the names are pretty much random, just whatever popped into my mind. The final sample tree someday will be considerably more sensible.)

Attachments:

[Uncle Buddy]

Last night I almost couldn't read what I was typing so I just settled for expressing the general idea and posted it. Today I can almost see, and will be able to see better after spending some time in the garden. At that time the attack on the families table will begin.

But I won't just delete all the code as I was threatening to do last night. I looked into my "superceded" folder and it looks like this is the third or fourth draft of the table, and it's already working pretty well. The GUI is likeable enough as is. Starting from scratch would be too much work under the circumstances.

Terminology:
parents: current person's biological parents, adoptive parents, foster parents, and guardians
progeny: current person's partners correlated with the children of each partnership

Goals:
I'll focus on replacing the master collection that builds the table. This large structure is causing me to go through unnecessary nesting of references due to its size and complexity. I realized there's no reason for the parents and progeny section of the families table to be created out of the same collection, since the two sections of the table don't reference each other in any way, and the two parts of the master collection are completely separate from each other.

With the master collection compounded together the way it is, instead of simply getting a reference something like this...

self.progeny[partner_id]
self.parents[0]["mother"]
self.parents[2]"adoptive father"]

...I've been forced to do something more like this:

self.families_data[1]...err...ummm.... gee whiz, hang on while I look this up....


Yep, after working on this feature for months, it's too complex to remember. I still have to look up the structure every time. So starting instead with a collection for parents and a separate collection for progeny, there will be less nesting in the references, so reading, writing, debugging and extending the code will be more intuitive.

Steps to take:
1) Replace the master collection with two smaller ones, then refactor the existing code to use these collections. It will be brought up to current working condition, which is not perfect, but some problems might incidentally get wiped out just by replacing the confusing stew of collected data.

2) The alt parent namings have to be cleaned up, because at some point I combined what had been separate alt parent and biological parent code. This slimmed up the code but introduced some glitches. Since I failed to rename everything properly, and my memory isn't very good, there are now some spots where the naming of things leaves it unclear as to what's supposed to be happening.

3) Get the Add Child feature working. This is the feature I haven't started yet, during the first few drafts of the families table.

4) Rewrite anything else that isn't working well.

The end result will be equivalent to starting over from scratch while simply plugging in the parts that aren't problematic such as the GUI and whatever easy logic code is obviously working right.

[Uncle Buddy]

On second thought, there's a definite tendency to dabble in the details when not starting over from scratch. It's a hard decision to make, a lot of thinking will have to happen any way you look at it, before this families table is able to be finished properly. Right now, so many changes have been made recently all over the app that I'm finding new flaws in the functionalities of the families table with each non-productive minute spent trying to rescue bad code from its deserved fate. A huge benefit of starting over with a blank slate is that it forces me to think but doesn't force me to think about something that's broken, so instead I think about the goal. Sometimes this is a big advantage. It's like, "You can't see the forest for the trees," but after a clearcut, you can see just fine.

[Uncle Buddy]

OK, I did it. I deleted all the code in families.py, GUI and all. When confusion becomes the main point, the goal can't be seen.

I'll add one person autofill and when that works right and does everything it's supposed to do, I'll add something else.

This is kinda extreme but recent interruptions have frozen the families table in a bad state and it's not gonna untangle itself.

This is gonna be fun. Then it will get complex again. Then I'll plough through, and finish it this time.

I have to watch out for shortcuts along the way that might lead to complexity down the road. Everything should be thought out and planned. I seldom plan anything, so in lieu of that, everything has to be taken one step at a time.

[Uncle Buddy]

May 28, 2022 22:14:37 GMT -8 Uncle Buddy said:

I'll add one person autofill and when that works right and does everything it's supposed to do, I'll add something else.

In this discussion of autofills, "used" means I'm typing along in an autofill, a matching string or "hit" fills in, and I accept or "use" the hit by tabbing out of the autofill which causes the accepted hit to be stored in the database.

The problem I was having is about the special functionality of simpler Treebard autofills which move a used hit to the front of the list. This causes the most recently-used hit to autofill first. This is very handy with places since they're so long and we do tend to re-use the same place over and over, for example while inputting every person, attribute and event on a single document such as census. It works for the event autofill but is not very important there, in fact using autofill there at all is not that important. I mainly use autofill for event types on the events table because otherwise I'd have to use comboboxes which would uglify the table. I want the table to look simple in spite of the complexity behind it. Since autofills have to be used for place fields, I use autofill on all the table cells for the sake of symmetry, but with most columns `autofill=False`. I turned autofill on for the events column to avoid using unwanted comboboxes there. The reason for having to use one or the other is so that an event type can be easily changed. For example, it would be easy to change an "occupation" event to a "career" attribute if you wanted to. (But impossible to change a "divorce" event to a "residence" event since they're not both couple events.)

Places and event types store their potential hits in a list. It's easy to move a hit that's just been used to the front of the list.

But person autofills now use a dictionary to store names in. This is a recent huge improvement which broke the "most recently used" feature of the person autofills. There are potentially a variety of names for any individual. The person ID has to be associated with the names, so it serves as the dict key. There have been at least two unproductive attempts to send a just-used key to the front of the list. I thought it might work since the Python dict now preserves the order in which its keys were created. But I stopped this time when the frustration reached a certain point, and asked myself, why am I even doing this? A person name isn't generally used over and over again in quick succession like a place name so often is. It could be, but with persons it's not the general trend. So that feature is gone. Person autofills now fill in according to the order in the dict, which is alphabetical I suppose, not in the order that a person hit was last used.

May 28, 2022 22:14:37 GMT -8 Uncle Buddy said:

OK, I did it. I deleted all the code in families.py, GUI and all.

And then proceeded to paste most of it back in, a few attributes at a time. As I did so, I became considerably more familiar with the code after our recent vacation from each other. The code flow became much more apparent to me. I preserved the GUI as is.

The main task, which was easy, was to split the big master collection `self.family_data` into two separate collections, `self.parents_data` and `self.progeny_data`. A number of sloppy details are being cleaned up, making the code easier to understand, appreciate, read, and extend. This families table is a very complex feature, unique to Treebard, and in general it already works. And a little voice inside me said, "Oh, by the way, if you try to rewrite this whole module from scratch, I quit." I don't know who that was talking to me but I took it sort of semi-seriously, just in case.

May 28, 2022 22:14:37 GMT -8 Uncle Buddy said:

Everything should be thought out and planned. I seldom plan anything, so in lieu of that, everything has to be taken one step at a time.

Actually I was pleasantly surprised to find most of the code is pretty reliable. The glitches will be somewhat easier to iron out when the floor's been swept and the windows cleaned.

Fortunately, I guess, a new flaw in the overall structure of the app was revealed yesterday that I hadn't expected. Changing the current person by double-clicking a name in the families table still works fine, but after doing so, pressing CTRL+S to redraw the table (which isn't generally necessary but still must work), changes parts of the current person tab back to a previous current person. I'll spare you the details, but the course I've decided to take is to rewrite the portion of the code that decides who the current person is. Because trying to fix this problem led to no end of confusion. Which means that the problem doesn't need to be analyzed, the feature just needs to be rewritten. It could be having an undecipherable bad effect anywhere, so it needs to be done before I finish the troubleshooting on the families table.

It finally became clear that all I needed was a good old-fashioned goal. And that goal is: define the current person once, in one single place of the app that all other portions of the app have reference to, and then in every procedure that changes the current person, change it in that one primal location only.

The problem with doing this is that a large portion of events_table.py consists of functions in the general module-level namespace, and in order to give these functions access to the current person, the current person was being gotten in a jumbled-up variety of ways. After some investigation, it seems that the reason I took those functions out of the EventsTable class some time ago might have had something to do with the fact that at one time I was instantiating the EventsTable class as separate events tables and attributes tables. Since those tables have now been combined into a single table, the reason for taking those functions out of the class appears to no longer be valid. So the plan is to get easier access to a centrally-created current person ID by not having the value jump through a bunch of hoops and perform a bunch of stupid pet tricks in order to define itself.

It looks like the right place to get current_person will be in main.py. The rest of the details should reveal themselves simply enough once I've got all those former methods stuffed back into that class where they belong, so that any function that needs a reference to current_person can do so without breaking a sweat.

[Uncle Buddy]

That was easy to do. EventsTable class is much simpler now and the current_person ID no longer thinks it needs to be initialized first in events_table.py. The original problems didn't magically resolve themselves; CTRL+S is still freakishly reverting everything to a former current person. But I will now be able to simplify the procedure for accessing current_person and get everything back into working order.

Another big change that's taken place recently was the redraw() feature which is called when new data is added that needs to be displayed on the person tab. The current person area at top, the families table, and the events table have to be redrawn, but this is now done by three separate methods in widgets.py which used to be a single method in events_table.py that was relatively inaccessible. The only trouble I was having with that feature, which is also run (but slightly differently) by pressing CTRL+S, has to do with a convoluted means of getting current_person right. Somewhere along the line, I lost interest in how that worked and let it get messed up. That's what's wrong with convoluted means of getting things done. It seems clever at the time, and might have been the culmination of blood, sweat and tears, but that don't make it good code.

So the way is clear now for finally getting current_person sitting on a very clear throne where all the pilgrims from all over the app can easily find it.

[Uncle Buddy]

The renovation of the logic code behind the families table is proceeding well, one feature at a time. The Add Child button works great, as far as I can tell so far. It's often easier to code something like this that sorta does one thing on the press of a button, but so often what I'm doing, including most of the fields in the families table, is to check a bunch of conditions and do several things when the user tabs out of an edited field, including maybe opening one or more dialogs to get or give more information. With quick focus-out edits and no unnecessary dialogs, Treebard is quick and easy to use. That doesn't make the code behind it easy to write.

The worst thing that can happen might be for multiple dialogs to open for the same attempted input. That sort of thing is to be avoided. It's not only annoying to the user, it's hard to keep the code from getting frantic. But when I set the bar higher, I get the expected complex problems to solve. I do OK for a novice coder but it takes a long time.

The current case for example is the totally normal-sounding case where the user adds a partner's name to a blank partner field, like a spouse, etc. Because Treebard, unlike Brand X genieware, doesn't decide exactly what the user means to do with any existing couple events and children, a focus-out event after adding a name to a partner field has to juggle interweaving sets of possibilities. Is the added partner an existing person or a new person? Of the marital events and children which the two partners share, which ones does the user actually want linked to the new partner? It still remains to check for the case where an existing partner is changed to a different person. This used to work, but who knows if it still does.

The good news is that the code isn't bad at all, now that some obvious noise has been smoothed out of it, and now that I've taken it apart one piece at a time and put it back together, refamiliarizing myself with its innards. Hopefully this will be the last rewrite before the families table manages to become fully functional. I'd love to move on to the next feature, but taking a break in the middle of a complicated feature like this can be disastrous, it can really slow things down.

[Uncle Buddy]

Here's an attempt to outline a problem wherein multiple marital events are being linked to the partner being added, with input needed from the user to do this, while at the same time the new partner is not only being added as a partner but is being added to the tree for the first time also.

Possible solutions:
1) Make it impossible to check more than one check box on the "select events to link" dialog if starting off with a blank partner field; (NO--it restricts user)
2) Don't let the second+ events show all at once, just show them one at a time; (OK but bloaty.)
3) Replace the error message with one that's more specific which instructs user to link to events one at a time when replacing a null partner, and to avoid this in the future, don't use null partners; (NO--it makes the user work & think harder instead of working harder myself to solve the problem by design.)
4) Stop allowing null partners altogether. (NO--can't live without blank partner fields.)
5) Solve the problem in such a way that the user can't tell the difference between this case and similar-but-different cases. The problem is that something is changing partway through a procedure, and this causes the same error message to open twice, besides causing plenty of confusion to the person trying to solve the problem. The data is currently going in correctly, but the user is confused and has to get lucky & do the right thing (he has to think, has to memorize a workaround). The solution is to smooth the existing fix for the user since it is now working. (BEST)

I choose #5, I plan to make sure the person-creating procedure finishes first and the event-linking procedure runs separately after that. If that's not good enough, the goal will have to be redefined.

[Uncle Buddy]

Easily solved with morning eyes. At the end of a long day of coding, I'm less likely to be able & willing to follow problematic code flow (i.e. read the code and understand what/where/when/why/how it does what it does). Choosing #5 above was the right thing to do, and when I spent about 5 or 10 minutes this morning following the code flow, I realized I was running something in a loop that should run only once. As a result, the Add Person dialog was running twice when I was only adding one person. I didn't notice this problem when I was only making one marital event and then linking it to a new person/added partner. I noticed it when I tried adding two marital events and linking them both to the same partner who was a new person I'd just created. All I had to do was lift a block of code out of one spot and plop it down in a different spot, no other changes. Adding partners to a blank partner field now works right.

The next step is to test the GUI's ability to unlink partners from specific events and children when the user deletes a name from a partner field. This used to work and maybe it still does. After that, I'll test the GUI's ability to change an existing partner to a different person, either a person that exists in the database or a new person just being added to the tree.

(Looks like unlinking on delete doesn't work right anymore, if it ever did. I can't remember. Looks like the other feature still works fine.)