Conversion Data Issues
This page lists all the problems with the data that were encountered during the data conversion process, and how the issue was resolved.
The problems are numbered, in the order in which they were encountered during the conversion.
Given a choice, earlier problems should be solved before later problems. This allows the later steps in the conversion to receive "correct" data, which help eliminate spurious problems, and aids the discovery of problems hidden by bad data.
Unsolved problems are marked with an *.
(#1) FOLLOW_MAP_TIME duplicate keys
Problem
The data dump says that the FOLLOW_MAP_TIME table has a primary key consisting of, in order, the columns: FMT_FOL_date, FMT_FOL_B_focal_AnimID, FMT_time.
But these columns contain duplicate values.
Bad Data
The duplicate values can be listed (from the raw schema) with:
This query no longer reports results because the data was changed in the original MS Access data.
SELECT *
FROM "FOLLOW_MAP_TIME"
JOIN (SELECT "FMT_FOL_date" AS the_date
, "FMT_FOL_B_focal_AnimID" AS the_animid
, "FMT_time" AS the_time
FROM "FOLLOW_MAP_TIME"
GROUP BY "FMT_FOL_date", "FMT_FOL_B_focal_AnimID", "FMT_time"
HAVING count(*) > 1
) AS fmt
ON ("FOLLOW_MAP_TIME"."FMT_FOL_date" = fmt.the_date
AND "FOLLOW_MAP_TIME"."FMT_FOL_B_focal_AnimID" = fmt.the_animid
AND "FOLLOW_MAP_TIME"."FMT_time" = fmt.the_time);
Solution
Fixed in MS Access 11/17/23 by ICG. Checked all against Tikis. When run in SokweDB, table does not exist
(#2) SUBADULT_ARRIVALS_LOG has a textual SA_first_tiki_date column
Problem
The SUBADULT_ARRIVALS_LOG table has a column that is supposed to contain a date, but instead contains the string "TEXTY".
Likely, the entire row is bad. The row contains:
SA_B_AnimID | SA_first_tiki_date | SA_notes
-------------+--------------------+----------
TEXTY | TEXTY | TEXTY
Bad Data
In the raw schema run:
This query no longer reports results because the data was changed in the original MS Access data.
select * from "SUBADULT_ARRIVALS_LOG" where "SA_first_tiki_date" = 'TEXTY';
Solution
Fixed in MS Access, 11/16/2023, ICG. Deleted row
(#3) BRECORD_NOTES contains rows where BREC_FOL_date has values that are not just a date
Problem
There are 4 rows in BRECORD_NOTES where the BREC_FOL_date column, supposedly a date, contains time values that are not '00:00:00'.
Bad Data
In the raw schema run:
This query no longer reports results because the data was changed in the original MS Access data.
select * from "BRECORD_NOTES" where "BREC_FOL_date"::TIME <> '00:00:00';
Solution
Fixed in MS Access, 11/16/2023, ICG Deleted time stamp
(#4) MATING_EVENT contains rows where M_FOL_date has values that are not just a date
Problem
The MATING_EVENT table contains 1 row where the time portion of M_FOL_date is not '00:00:00'.
Bad Data
In the raw schema run:
This query no longer reports results because the data was changed in the original MS Access data.
select * from "MATING_EVENT" where "M_FOL_date"::TIME WITHOUT TIME ZONE <> '00:00:00';
Solution
Fixed in MS Access, 11/16/2023, ICG Deleted time stamp
(#5) BIOGRAPHY.DepartdateError data discarded
Problem
The BIOGRAPHY.DepartdateError column contains data that was, after discussion, determined to be unusable.
Bad Data
In the raw schema run:
This query no longer reports results because the data was changed in the original MS Access data.
select * from raw."BIOGRAPHY" where "B_DepartdateError" <> 0;
Solution
Do not convert the data. There is no corresponding column in the new db design.
(#6) BIOGRAPHY.B_AnimID_num column contains the empty string
Problem
The BIOGRAPHY.B_AnimID_num column contains the empty string, instead of NULL.
Bad Data
In the raw schema run:
This query no longer reports results because the data was changed in the original MS Access data.
select * from raw."BIOGRAPHY" where "B_AnimID_num" = '';
Solution
Changed 11 empty string values to NULL in MS Access. ICG 12/6/2023
(#7) BIOGRAPHY.B_AnimID_num column is textual
Problem
The BIOGRAPHY.B_AnimID_num column contains has a data type of TEXT. The data values all begin with "CH".
Bad Data
In the raw schema run:
This query no longer reports results because the data was changed in the original MS Access data.
select "B_AnimID", "B_AnimID_num"
from raw."BIOGRAPHY"
where "B_AnimID_num" <> ''
AND "B_AnimID_num" IS DISTINCT FROM NULL
order by "B_AnimID";
Solution
Removed the "CH" prefix and made the column an integer in MS Access. ICG 12/6/2023
(#8) BIOGRAPHY.DadID_publication_info column contains NULL values
No longer a problem as NULLs are now allowed.
(Not) Problem
The BIOGRAPHY.DadID_publication_info column has a data type that allows NULL values.
SokweDB wants only text
Bad Data
In the raw schema run:
This query no longer reports results because the data was changed in the original MS Access data.
select * from raw."BIOGRAPHY" where "B_DadID" IS NULL order by "B_AnimID";
Solution
Changed the NULL to the empty string ("") in MS Access. ICG 12/6/2023
(#9) BIOGRAPHY.DadID column contains non-AnimID values
Problem
The BIOGRAPHY.DadID column has a non-AnimID values.
Bad Data
In the raw schema run:
select "B_AnimID", "B_DadID"
from raw."BIOGRAPHY"
where "B_DadID" is not null
and "B_DadID" <> ''
and not exists (select 1
from raw."BIOGRAPHY" as search
where search."B_AnimID" = raw."BIOGRAPHY"."B_DadID");
Solution
Create the DadIDPrelim column in a BIOGRAPHY_DATA table, and make a BIOGRAPHY view that combines Dad_ID and DadIDPrelim into DadID -- adding the '_prelim' suffix as expected.
(#10) BRECORD_NOTES contains rows where BREC_time has values that are not just a time
Problem
There are 56 rows in BRECORD_NOTES where the BREC_time column, supposedly a time, contains date values that are not '1899-12-30'.
Bad Data
In the raw schema run:
select * from raw."BRECORD_NOTES" where "BREC_time"::DATE <> '1899-12-30';
Solution
The data is fixed in the conversion process.
(#11) KAZ has b_dadid_publication_info, but b_dadid is NULL
Problem
KAZ has dad id publication info of 'Rudicell et al. 2010', but a NULL dadid.
Bad Data
In the clean schema run:
select *
from clean.biography
where b_dadid is NULL
and (b_dadid_publication_info <> ''
or b_dadid_publication_info is null);
Solution
KAZ has 2 potential dads. Introduce a DadIDStatus column, to replace the DadIDPrelim column and have a code that describes what's going on with KAZ.
Fixed in the MS Access data; KAZ was assigned the UNK individual as the dad. Problem will be marked resolved with the upload of the next MS Access database dump.
(#12) 9 BIOGRAPHY rows have BirthComm values that are not COMM_IDS.CommID values
Problem
There are 9 BIOGRAPHY rows with BirthComm values that are not valid COMM_IDS, their communities do not exist.
Bad Data
In the clean schema run:
select *
from easy.biography
where b_birthgroup is not NULL
and not exists (select 1
from easy.community_lookup
where community_lookup.cl_community_id
= biography.b_birthgroup);
Solution
Added KL and KL_KK to CommIds. KL = Kalande, KL_KK= Kasekela/kalande codes are created during conversion
Fixed in commit 5d85b83bfe1a.
(#13) COMM_MEMBS rows place individuals in a community, that is not their birth community, before their BIOGRAPHY.EntryDate
Problem
13 COMM_MEMBS rows place individuals into a community, that is not their birth community, before BIOGRAPHY says they entered the community.
update: 12 rows as of 2026-07-19
Bad Data
In the clean schema run:
select b.b_animid, b.b_birthgroup, b.b_entrydate, cm.cm_start_date
from clean.community_membership as cm
join clean.biography as b
on (b.b_animid = cm.cm_b_animid)
where b.b_birthgroup is distinct from cm.cm_cl_community_id
and cm.cm_start_date < b.b_entrydate
order by b.b_animid, cm.cm_start_date;
Solution
Talk to Karl about kk_P0 and kk_P1
(#14) COMM_MEMBS rows place individuals in a community after their BIOGRAPHY.EndDate
Problem
3 COMM_MEMBS rows place individuals into a community after BIOGRAPHY says they left the community.
Bad Data
In the clean schema run:
select b.b_animid, cm.cm_end_date
from clean.community_membership as cm
join clean.biography as b
on (b.b_animid = cm.cm_b_animid)
where cm.cm_end_date > b.b_departdate
order by b.b_animid, cm.cm_end_date;
Solution
ICG EVL fixed BH, HAI, TZB2 in Access. Need to talk to Karl about MG, RO, WD. Treat like KL chimps in Biography?
(#15) TT is placed in a community twice on the same day
Problem
An individual may not be in more than one community (or even twice in the same community) on any given day.
Bad Data
In the clean schema run:
select first.cm_b_animid as anim_id
, first.cm_start_date as first_start_date
, first.cm_end_date as first_end_date
, first.cm_cl_community_id as first_community_id
, first.cm_start_source as first_start_source
, first.cm_end_source as first_end_source
, second.cm_start_date as second_start_date
, second.cm_end_date as second_end_date
, second.cm_cl_community_id as second_community_id
, second.cm_start_source as second_start_source
, second.cm_end_source as second_end_source
from clean.community_membership as first
join clean.community_membership as second
on (first.cm_b_animid = second.cm_b_animid
and first.cm_start_date < second.cm_start_date)
where first.cm_end_date >= second.cm_start_date;
Solution
Fixed in MS Access 2/1/2024 Changed end date of KK_P1 membership to 8/14/2022
ICG
(#16) There are 31 rows in COMM_MEMB_LOG where MadeBy is NULL
Problem
There are 31 rows in COMM_MEMB_LOG where MadeBy is NULL, and the column does not allow NULL values.
Bad Data
In the clean schema run:
select * from clean.community_membership_update_log where made_by is null;
Solution
See problem #18. Fixed in commit 7a48fb2c4aad3b5bd4
(#17) There are 64 rows in COMM_MEMB_LOG where the chimp_id is not on BIOGRAPHY
Problem
There are 64 rows in COMM_MEMB_LOG where the chimp_id is not a BIOGRAPHY.AnimID.
Bad Data
In the clean schema run:
select date_of_update, chimp_id
from clean.community_membership_update_log as log
where not exists (select 1
from clean.biography
where biography.b_animid = log.chimp_id);
Solution
Make this a "soft" error. Fixed in commit 4c9626b304.
(#18) There are 31 rows in COMM_MEMB_LOG where the MadeBy is NULL
Problem
There are 31 rows in COMM_MEMB_LOG where MadeBy is NULL, and the column does not allow NULLs.
Bad Data
In the clean schema run:
select * from clean.community_membership_update_log where made_by is null;
Solution
Create an unknown person (UNK), and when the person is NULL, use the unkonwn person.
This allows us to make the unknown person "inactive", preventing them from being used in newly entered data. The alternative, allowing NULL MadeBy values, would allow new "bad data", and NULL values make querying harder.
Fixed in commit 7a48fb2c4aad3b5b.
(#19) There are 18 rows in BIOGRAPHY_LOG where the MadeBy is NULL
Problem
There are 18 rows in BIOGRAPHY_LOG where MadeBy is NULL, and the column does not allow NULLs.
Bad Data
In the clean schema run:
select * from clean.biography_update_log where made_by is null;
Solution
Create an unknown person (UNK), and when the person is NULL, use the unkonwn person. (See Problem #18)== * (#19) There are 18 rows in BIOGRAPHY_LOG where the MadeBy is NULL ==
Fixed in commit 8a528cc7d1a9fb.
update 2026-07-26:
The 18 problematic values are zero-length strings, not NULL, so they pass through the ELSE made_by branch and violate the foreign key as madeby=(). This solution was hardened with commit 6279986f5f77cab69d19733925f173eadca23b60.
(#20) There are 3 rows in BIOGRAPHY_LOG where the Rationale is NULL
Problem
There are 3 rows in BIOGRAPHY_LOG where Rationale is NULL. The column does not allow NULLs; normally the conversion program would convert NULL to the empty string. But the Rationale column requires there be (non-empty) textual data.
Bad Data
In the clean schema run:
select * from clean.biography_update_log where update_rationale is null;
Solution
ICG fixed in MS Access 2/8/2024. Updated rationale to 'routine update'.
(#21) There are 6 rows in BIOGRAPHY_LOG where the update_escription is NULL
Problem
There are 6 rows in BIOGRAPHY_LOG where Description is NULL. It appears that the description was put into the Rationale column, sometimes along with some rationale.
Bad Data
In the clean schema run:
select * from clean.biography_update_log where update_description is null;
Solution
ICG fixed in MS Access 2/8/2024, using information in update_rationale
(#22) There are 85 rows in BIOGRAPHY_LOG where the chimp_id is not on BIOGRAPHY
Problem
There are 85 rows in BIOGRAPHY_LOG where the chimp_id is not a BIOGRAPHY.AnimID.
Bad Data
In the clean schema run:
select *
from clean.biography_update_log as log
where not exists (select 1
from clean.biography
where biography.b_animid = log.chimp_id);
Solution
Make this a "soft" error. Fixed in commit c96555f9f326.
(#23) There are 65 rows in BIOGRAPHY_LOG where the MadeBy is not on PEOPLE
Problem
There are 65 rows in BIOGRAPHY_LOG where the MadeBy is not on PEOPLE, but there is MadeBy data.
These are "combination" ID errors, where multiple people are entered instead of a single people code.
Bad Data
In the clean schema run:
select *
from clean.biography_update_log as log
where not exists (select 1
from clean.people
where people.person = log.made_by)
and made_by is not null;
Solution
Fixed in MS Access by ICG 2/8/2024. Changed multiple IDs to the one who made the final change.
(#24) Follow starts are not first arrivals
Problem
There are 6,397 cases where the old FOLLOW table's columns "fol_time_begin" is not the first arrival time, fa_time_start, on FOLLOW_ARRIVAL.
The The Gombe Chimpanzee Database Handbook says that this should never happen, because the follow start is supposed to be the first arrival. Is there additional data in the old fol_time_begin, like the actual start the observers started working? If not, which data is correct, the one in FOLLOW or the one in FOLLOW_ARRIVAL?
Bad Data
WITH spans AS
(SELECT fa_fol_date, fa_fol_b_focal_animid
, MIN(fa_time_start) AS min_start
FROM clean.follow_arrival
WHERE fa_b_arr_animid = fa_fol_b_focal_animid
GROUP BY fa_fol_date, fa_fol_b_focal_animid)
SELECT spans.*, follow.fol_time_begin
FROM spans
JOIN clean.follow
ON (follow.fol_date = spans.fa_fol_date
AND follow.fol_b_animid = spans.fa_fol_b_focal_animid)
WHERE NOT EXISTS
(SELECT 1
FROM clean.follow
WHERE follow.fol_date = spans.fa_fol_date
AND follow.fol_b_animid = spans.fa_fol_b_focal_animid
AND spans.min_start = follow.fol_time_begin)
ORDER BY spans.fa_fol_date, spans.fa_fol_b_focal_animid;
Solution
This is data entry error, ignore the problem and ignore the data in the follow.fol_time_begin.
(#25) Follow ends are not last arrivals
Problem
There are 6,570 cases where the old FOLLOW table's columns "fol_time_end" is not the last arrival time, fa_time_end, on FOLLOW_ARRIVAL.
The The Gombe Chimpanzee Database Handbook says that this should never happen, because the follow end is supposed to be the first arrival/last departure. Is there additional data in the old fol_time_end, like the actual end-time the observers started working? If not, which data is correct, the one in FOLLOW or the one in FOLLOW_ARRIVAL?
Bad Data
WITH spans AS
(SELECT fa_fol_date, fa_fol_b_focal_animid
, MAX(fa_time_end) AS max_end
FROM clean.follow_arrival
WHERE fa_b_arr_animid = fa_fol_b_focal_animid
GROUP BY fa_fol_date, fa_fol_b_focal_animid)
SELECT spans.*, follow.fol_time_end
FROM spans
JOIN clean.follow
ON (follow.fol_date = spans.fa_fol_date
AND follow.fol_b_animid = spans.fa_fol_b_focal_animid)
WHERE NOT EXISTS
(SELECT 1
FROM clean.follow
WHERE follow.fol_date = spans.fa_fol_date
AND follow.fol_b_animid = spans.fa_fol_b_focal_animid
AND spans.max_end = follow.fol_time_end)
ORDER BY spans.fa_fol_date, spans.fa_fol_b_focal_animid;
Solution
This is data entry error, ignore the problem and ignore the data in the follow.fol_time_end.
(#26) Mismatch of start-in-nest on follow and follow_arrival
Problem
There are 1,848 3,369 follows where the follow arrival says the focal started in the nest but the follow does not, or vice-versa.
Bad Data
WITH spans AS
(SELECT fa_fol_date, fa_fol_b_focal_animid
, MIN(fa_time_start) AS min_start
FROM easy.follow_arrival
WHERE fa_b_arr_animid = fa_fol_b_focal_animid
GROUP BY fa_fol_date, fa_fol_b_focal_animid)
SELECT spans.*, follow.fol_time_begin, follow.fol_flag_begin_in_nest
, first_arrivals.fa_type_of_nesting
FROM easy.follow
JOIN spans
ON (follow.fol_date = spans.fa_fol_date
AND follow.fol_b_animid = spans.fa_fol_b_focal_animid)
JOIN easy.follow_arrival
AS first_arrivals
ON (first_arrivals.fa_fol_date = follow.fol_date
AND first_arrivals.fa_fol_b_focal_animid = follow.fol_b_animid
AND first_arrivals.fa_time_start = spans.min_start)
WHERE (((first_arrivals.fa_type_of_nesting = 1
OR first_arrivals.fa_type_of_nesting = 3)
AND follow.fol_flag_begin_in_nest = 0)
OR (first_arrivals.fa_type_of_nesting <> 1
AND first_arrivals.fa_type_of_nesting <> 3
AND follow.fol_flag_begin_in_nest = 1))
ORDER BY spans.fa_fol_date, spans.fa_fol_b_focal_animid;
Solution
Ignore. Default to data from Follow_Arrival
Remarks
When applying the solution, solving both problem #26 and #27, and updating follow_arrival so that it is the one source of truth used by the conversion, there are 297 follow_arrival rows updated.
This implies that it is primarily the follow table that does not mark the individual as being in a nest when they should be in a nest. ("Should be", according to the accepted solution.)
(#27) Mismatch of end-in-nest on follow and follow_arrival
Problem
There are 3,804 follows where the follow arrival says the focal ended in the nest but the follow does not, or vice-versa.
Bad Data
WITH spans AS
(SELECT fa_fol_date, fa_fol_b_focal_animid
, MAX(fa_time_end) AS max_end
FROM easy.follow_arrival
WHERE fa_b_arr_animid = fa_fol_b_focal_animid
GROUP BY fa_fol_date, fa_fol_b_focal_animid)
SELECT spans.*
, follow.fol_time_end, follow.fol_flag_end_in_nest
, last_arrivals.fa_type_of_nesting
FROM easy.follow
JOIN spans
ON (follow.fol_date = spans.fa_fol_date
AND follow.fol_b_animid = spans.fa_fol_b_focal_animid)
JOIN easy.follow_arrival
AS last_arrivals
ON (last_arrivals.fa_fol_date = follow.fol_date
AND last_arrivals.fa_fol_b_focal_animid = follow.fol_b_animid
AND last_arrivals.fa_time_end = spans.max_end)
WHERE (((last_arrivals.fa_type_of_nesting = 2
OR last_arrivals.fa_type_of_nesting = 3)
AND follow.fol_flag_end_in_nest = 0)
OR (last_arrivals.fa_type_of_nesting <> 2
AND last_arrivals.fa_type_of_nesting <> 3
AND follow.fol_flag_end_in_nest = 1))
ORDER BY spans.fa_fol_date, spans.fa_fol_b_focal_animid;
Solution
Ignore the problem. Default to Follow_Arrival
Remarks
See the remarks for problem #26.
(#28) The FOLLOW.FOL_distance_traveled column is not converted
Problem
The FOLLOW.FOL_distance_traveled column has no corresponding column in the new database design. The conversion process does not check that the value of this column is consistent with the other data in the database from which it is computed.
Solution
This is a computed column and does not need to be converted. The desired value is computed in the design of the new database.
The assumption is that the "raw" data from which this value is computed in the MS Access database is correct.
(#29) The FOLLOW.Brecord_notes column is not converted
Problem
The FOLLOW.Brecord_notes column has no corresponding column in the new database design.
Solution
This column is used for administrative purposes and does not need to be in the new database design.
(#30) Some follows have no community
Problem
There are 13 follows with no community.
Bad Data
select * from clean.follow where fol_cl_community_id is null order by fol_date, fol_b_animid;
solution
Solved
(#31) Some follows have an animid with trailing spaces
Problem
There are 9 39 follows with animids that don't exist, because they have trailing spaces. These are comprised of 9 23 distinct animids.
Bad Data
Cleaned up in clean schema, so query easy schema.
select rtrim(fol_b_animid) from easy.follow where rtrim(fol_b_animid) <> fol_b_animid order by fol_b_animid;
Solution
Remove the trailing spaces in the conversion process.
(#32) Some follows have an animid that is lower-case
Problem
There is 1 follows with an animids that don't exist, because it is lower-case.
Bad Data
Cleaned up in clean schema, so query easy schema.
select * from easy.follow where upper(fol_b_animid) <> fol_b_animid order by fol_b_animid;
Solution
Convert the animid to upper-case in the conversion process.
(#33) Some follows have an animid that does not exist, even after animid cleanup
Problem
There are 14 follows with animids that don't exist, even after cleanup that removes trailing spaces and forces upper-case. (Some of these may be due to prior errors.) These are comprised of 3 distinct animids.
Bad Data
-- The 14 follows with bad animids
select *
from clean.follow
where not exists
(select 1
from clean.biography
where biography.b_animid = upper(rtrim(follow.fol_b_animid)))
order by fol_date, fol_b_animid;
-- The 3 animids involved
select distinct follow.fol_b_animid
from clean.follow
where not exists
(select 1
from clean.biography
where biography.b_animid = upper(rtrim(follow.fol_b_animid)))
order by follow.fol_b_animid;
Solution
Solved
(#34) There are duplicate animid, date combinations on FOLLOW
Problem
There are 2 sets of duplicate animid, date combinations on the FOLLOW table, for a total of 4 rows.
Bad Data
-- The duplicate animid, date combinations select follow.fol_b_animid, follow.fol_date, count(*) from clean.follow group by follow.fol_b_animid, follow.fol_date having count(*) > 1 order by fol_b_animid, fol_date;
-- The duplicate rows
with dups as (
select follow.fol_b_animid, follow.fol_date
from clean.follow
group by follow.fol_b_animid, follow.fol_date
having count(*) > 1)
select *
from clean.follow
join dups on (follow.fol_b_animid = dups.fol_b_animid
and follow.fol_date = dups.fol_date)
order by follow.fol_b_animid, follow.fol_date;
Solution
Solved
(#35) There are follows done before a focal was under study
Problem
There are 5 follows that are done before their focal was under study, before the focal's EntryDate.
Bad Data
select follow.*, biography.b_entrydate
from clean.follow
join clean.biography on (biography.b_animid = follow.fol_b_animid)
where biography.b_entrydate > follow.fol_date
order by follow.fol_date, follow.fol_b_animid;
Solution
Solved
(#36) FOLLOW_OBSERVERS.Period is not checked against follow start or stop times
Problem
The conversion process uses the clean.follow columns of fol_am_observer_1, fol_am_observer_2, fol_pm_observer_1, and fol_pm_observer_2 to populate FOLLOW_OBSERVERS. (The *_observer_1 column going into FOLLOW_OBSERVERS.OBS_BRec and the *_observer_2 column going in OBS_Tiki.)
If both the *_observer_1 and the *_observer_2 columns are either NULL or the empty string (after space trimming), the no row is created for the respective time period.
There are no checks done to ensure that the time periods of the follow have any relation to the FOLLOW_OBSERVERS.Period value.
Solution
Ignore the problem. (If someone cares, add a query to the warning system.)
(#37) Some follows have no recorded observers
Problem
There are 56 follows with no recorded observers, but the system requires there be a FOLLOW_OBSERVERS record related to the follow.
See problem #36 for a description of the follow observer conversion process.
Bad data
select *
from clean.follow
where coalesce(btrim(fol_am_observer_1), '') = ''
and coalesce(btrim(fol_am_observer_2), '') = ''
and coalesce(btrim(fol_pm_observer_1), '') = ''
and coalesce(btrim(fol_pm_observer_2), '') = ''
order by fol_date, fol_b_animid;
Solution
Make the "NONE" (no observer) person the observers.
Create a "UNK" (unknown) time period in PERIODS, and make that the time period.
(#38) Some follows have only one observer, but the system wants two
Problem
There are follows with only one observer, but the system requires a FOLLOW_OBSERVERS record have a value in both OBS_BRec and OBS_Tiki.
See problem #36 for a description of the follow observer conversion process.
Solution
Create a "NONE" person, and make that person the observer when there's otherwise not a value.
(#39) In follow, there are observers that are 2 people
Problem
In the follow table, in columns fol_am_observer_1, fol_am_observer_2, fol_pm_observer_1, fol_pm_observer_2, there are 30 rows (28, really, because of n/a) that appear to represent 2 people.
These look like names, separated by the "/" character.
Bad data
WITH new_people AS (
SELECT BTRIM(follow.fol_am_observer_1) AS person
FROM clean.follow
WHERE follow.fol_am_observer_1 IS NOT NULL
GROUP BY BTRIM(follow.fol_am_observer_1)
UNION
SELECT BTRIM(follow.fol_am_observer_2) AS person
FROM clean.follow
WHERE follow.fol_am_observer_2 IS NOT NULL
GROUP BY BTRIM(follow.fol_am_observer_2)
UNION
SELECT BTRIM(follow.fol_pm_observer_1) AS person
FROM clean.follow
WHERE follow.fol_pm_observer_1 IS NOT NULL
GROUP BY BTRIM(follow.fol_pm_observer_1)
UNION
SELECT BTRIM(follow.fol_pm_observer_2) AS person
FROM clean.follow
WHERE follow.fol_pm_observer_2 IS NOT NULL
GROUP BY BTRIM(follow.fol_pm_observer_2)
)
, uniq_people AS (
SELECT new_people.person
FROM new_people
WHERE new_people.person <> ''
GROUP BY new_people.person
)
SELECT uniq_people.person
FROM uniq_people
JOIN uniq_people AS up
ON (LOWER(uniq_people.person) = LOWER(up.person))
WHERE uniq_people.person <> up.person
AND STRPOS(uniq_people.person, '/') <> 0
GROUP BY uniq_people.person
ORDER BY LOWER(uniq_people.person), uniq_people.person;
Solution
Ignore the problem. Mark all people containing a "/" character as not active, so they can't be used in the future.
(#40) In follow, there are observers that differ only by character case
Problem
In the follow table, in columns fol_am_observer_1, fol_am_observer_2, fol_pm_observer_1, fol_pm_observer_2, there are 511 names that differ only by character case. So, about half that in terms of unique names.
This is complicated to account for and exclude duplicates in the conversion process. So resolution of this is holding back additional conversion work.
Bad data
Note the use of the easy schema, instead of the
clean schema. This is because the observer data
in the clean schema has been case-normalized.
WITH new_people AS (
SELECT BTRIM(follow.fol_am_observer_1) AS person
FROM easy.follow
WHERE follow.fol_am_observer_1 IS NOT NULL
GROUP BY BTRIM(follow.fol_am_observer_1)
UNION
SELECT BTRIM(follow.fol_am_observer_2) AS person
FROM easy.follow
WHERE follow.fol_am_observer_2 IS NOT NULL
GROUP BY BTRIM(follow.fol_am_observer_2)
UNION
SELECT BTRIM(follow.fol_pm_observer_1) AS person
FROM easy.follow
WHERE follow.fol_pm_observer_1 IS NOT NULL
GROUP BY BTRIM(follow.fol_pm_observer_1)
UNION
SELECT BTRIM(follow.fol_pm_observer_2) AS person
FROM easy.follow
WHERE follow.fol_pm_observer_2 IS NOT NULL
GROUP BY BTRIM(follow.fol_pm_observer_2)
)
, uniq_people AS (
SELECT new_people.person
FROM new_people
WHERE new_people.person <> ''
GROUP BY new_people.person
)
SELECT uniq_people.person
FROM uniq_people
JOIN uniq_people AS up
ON (LOWER(uniq_people.person) = LOWER(up.person))
WHERE uniq_people.person <> up.person
GROUP BY uniq_people.person
ORDER BY LOWER(uniq_people.person), uniq_people.person;
Solution
Use the mixed case code when such exists. This involves changing the observer columns in the follow table. See the notes.
(#41)There are follow arrivals with NULL nesting information
Problem
In the follow_arrival table, there are 11 rows with a NULL fa_type_of_nesting.
Bad data
select * from clean.follow_arrival where fa_type_of_nesting IS NULL;
Solution
Solved
(#42) There are follow arrivals with NULL sexual cycle information
Problem
In the follow_arrival table, there are 222 rows with a NULL fa_type_of_cycle.
Bad data
select * from clean.follow_arrival where fa_type_of_cycle is null;
Solution
Make a MISS CYCLE_STATES value and use that for the arrivals with no cycle information.
Problem
In the follow_arrival table, there are 732 276 75 rows with a focal and a date that have no matching information on the follow table.
Bad data
SELECT *
FROM clean.follow_arrival
WHERE NOT EXISTS
(SELECT 1
FROM clean.follow
WHERE follow.fol_b_animid = follow_arrival.fa_fol_b_focal_animid
AND follow.fol_date = follow_arrival.fa_fol_date)
ORDER BY follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid;
Solution
5/5/2026 ICG FIXED MANY ROWS IN MS ACCESS
Most of the rest are hand-entered juvenile arrivals from Brec. Spot checks have Brec notes but no physical tiki Need to investigate further - brec swahili? Eventually the rest of the individuals in the Brec notes need to be entered by hand.
Should be solved by the WATCHES table
For now, allow these rows
Remarks on Solution
There is no follow to attach the arrival to. No WATCHES row. So there is no way to get the data in and simply write a warning.
Allowing these rows requires a re-design of the database and additional "hard" rules. Which is going to take time to program, and delay finishing. And, it will take additional time to rip-out the rules when you finally resolve the data issue.
What is needed is to make a new WATCHES.Type just to allow for arrivals with no related follow, and ensure that only arrivals without follows are attaching to this new Type.
The only "easy" alternative I can think of is:
Don't convert this data at this time. Convert it when we get around to fixing the data. In the mean time, anyone who wants to work with this data can get it from the "clean" schema, using the query provided in the problem #43 section, and figure out how to integrate it with whatever they are working on.
(#44) There are follow arrivals where the arriving chimp arrives before being under study
Problem
In the follow_arrival table, there are 217 49 rows where the fa_b_arr_animid, the arriving individual, has an entry date after the date of the follow.
Bad data
select follow_arrival.*, biography.b_entrydate, biography.b_sex
from clean.follow_arrival
join clean.biography
on (biography.b_animid = follow_arrival.fa_b_arr_animid)
where biography.b_entrydate > follow_arrival.fa_fol_date
order by biography.b_animid, follow_arrival.fa_fol_date;
Solution
Something is wrong with community entry date or biography
SN1 fixed in MS Access ICG 5/19/2026
Elo Confirmed CT and SG were with CA on 10/23/1980 from brec swahili
RESOLVED
(#45) The follow_arrival.fa_update column is not preserved
Problem
There is nowhere in the current design to store the values in the follow_arrival.fa_update column.
Solution
Add a temporal extension to Postgres to make the db into a temporal database to track change history and be able to see data as it existed at any point in time.
FOLLOWUP WITH KARL. Will re-review this should a temporal extension be out of budget, etc.
ASK KARL WHY THIS COLUMN CAN'T GO IN
There is now a ARRIVALS.Updated column. This provides a place to put the existing data. Going forward, a PG extension (see below) can be installed for more functionality.
Wouldn't you rather have a temporal database? (Simple description in the top paragraph here.) This would allow you to "time travel" and look at the database content at any point in time. And, it puts a "last changed" timestamp on every row, of every table you say to track temporally, automatically.
If so, install one of these extensions to PostgreSQL. (Or, as a separate project, I will improve one of them so that the people who don't need to time travel don't see any changes to the existing database at all. What I don't like about most of the extensions listed is that they add extra columns to existing tables, as well as adding extra "history" tables in places where they clutter up what users really need to see.)
(#46) There are follow_arrivals where non-females have a cycle code that is other than n/a
Problem
There are 169 168 follow_arrival rows, for non-female arriving individuals, that have a sexual cycle code (fa_type_of_cycle) that is not n/a.
Bad data
select follow_arrival.*, biography.b_sex
from clean.follow_arrival
join clean.biography
on (biography.b_animid = follow_arrival.fa_b_arr_animid)
where biography.b_sex <> 'F'
and follow_arrival.fa_type_of_cycle <> 'n/a'
order by follow_arrival.fa_fol_date
, follow_arrival.fa_b_arr_animid;
To summarize by sex and cycle code:
select biography.b_sex, follow_arrival.fa_type_of_cycle, count(*)
from clean.follow_arrival
join clean.biography
on (biography.b_animid = follow_arrival.fa_b_arr_animid)
where biography.b_sex <> 'F'
and follow_arrival.fa_type_of_cycle <> 'n/a'
group by biography.b_sex, follow_arrival.fa_type_of_cycle order by biography.b_sex
, follow_arrival.fa_type_of_cycle;
Solution
FIX THE FEW ACTUAL MALES THAT DON'T HAVE AN n/a
Change non-females (INCLUDING UNKNOWN SEX) who have a cycle code of 0 to a cycle code of n/a. IAN TO FIX IN ACCESS
IAN FIXED IN ACCESS 3/25/2026
* (#47) There are follow_arrivals where females that are too young have a cycle code of U
Problem
There are 16 follow_arrival rows, for female arriving individuals, that have a sexual cycle code (fa_type_of_cycle) of U but are less than 5 years of age.
Naturally, this number will change if the age limit is changed, but this is here as a placeholder.
Note: The test is against the birthdate, not the minimum possible birthdate.
Bad data
SELECT fa.*
, b.b_sex
, b.b_birthdate
, AGE(fa.fa_fol_date, b.b_birthdate) AS age_at_follow
, INTERVAL '5 years'
- AGE(fa.fa_fol_date, b.b_birthdate) AS younger_than_five_by
FROM clean.follow_arrival AS fa
JOIN clean.biography AS b
ON b.b_animid = fa.fa_b_arr_animid
WHERE b.b_sex = 'F'
AND fa.fa_type_of_cycle = 'U'
AND b.b_birthdate > fa.fa_fol_date - INTERVAL '5 years'
ORDER BY fa.fa_fol_date
, fa.fa_fol_b_focal_animid
, fa.fa_b_arr_animid;
Note: The original query used the wrong inequality symbol.
Solution
Changing the limit from 6 years of age to 5 reduced the number of outstanding problems to 16.
The rest will have to be fixed in the data. Alternately, we can adjust the hard limit, and set a soft limit of 5 years in the warning system with a note to change the hard limit back once the errors are resolved.
Resolution: Fixed in the data.
UPDATE 2026-07-23 it does not appear that these data were fixed, please see below for a summary of this issue and related Problem #49:
UPDATE 2026-08-14: fixed in Access by ICG
UPDATE 2026-08-19: improved but there are still 12 offending records
| problem | problem_description | age_years | current_cycle | youngest_exact_age | oldest_exact_age | offending_rows | problem_total |
|---|---|---|---|---|---|---|---|
| 47 | U assigned before age 5 | 1 | U | 1 year 4 mons 5 days | 1 year 4 mons 5 days | 1 | 21 |
| 47 | U assigned before age 5 | 2 | U | 2 years 11 mons 26 days | 2 years 11 mons 26 days | 1 | 21 |
| 47 | U assigned before age 5 | 3 | U | 3 years 3 mons 30 days | 3 years 11 mons 1 day | 10 | 21 |
| 47 | U assigned before age 5 | 4 | U | 4 years 5 mons 21 days | 4 years 9 mons 18 days | 9 | 21 |
| 49 | U assigned at age 15 or older | 15 | U | 15 years 1 mon 2 days | 15 years 10 mons 14 days | 16 | 38 |
| 49 | U assigned at age 15 or older | 16 | U | 16 years 19 days | 16 years 11 mons 6 days | 7 | 38 |
| 49 | U assigned at age 15 or older | 17 | U | 17 years 1 mon 19 days | 17 years 1 mon 19 days | 1 | 38 |
| 49 | U assigned at age 15 or older | 18 | U | 18 years 1 mon 18 days | 18 years 8 mons 16 days | 6 | 38 |
| 49 | U assigned at age 15 or older | 32 | U | 32 years 1 mon 21 days | 32 years 5 mons 14 days | 4 | 38 |
| 49 | U assigned at age 15 or older | 33 | U | 33 years 6 days | 33 years 6 days | 4 | 38 |
option matrix:
| Value | Meaning | Accepted under age 5? |
|---|---|---|
0 |
Not swollen | Yes |
MISS |
Missing data | Yes |
U |
Adolescent swelling | No |
0.25, 0.5, 0.75, 1 |
Increasing observed swelling | No—female must be at least 8 |
n/a |
Male/not applicable | No—prohibited for females |
* (#48) There are follow arrivals where the arriving chimp arrives after finishing being under study
Problem
In the follow_arrival table, there are 239 324 rows where the fa_b_arr_animid, the arriving individual, has a departure date before the date of the follow.
Bad data
select follow_arrival.*, biography.b_departdate, biography.b_sex
from clean.follow_arrival
join clean.biography
on (biography.b_animid = follow_arrival.fa_b_arr_animid)
where biography.b_departdate < follow_arrival.fa_fol_date
order by biography.b_animid, follow_arrival.fa_fol_date;
SOLUTION
IAN TO FIX IN ACCESS - CHECK DATES OF FOLLOWS AND ACCURACY OF IDS
* (#49) There are follow_arrivals where females that are too old have a cycle code of U
Problem
There are 127 98 86 48 follow_arrival rows, for female arriving individuals, that have a sexual cycle code (fa_type_of_cycle) of U but are more than 14 years of age.
(Actually, because the endpoint takes up the whole 14th year, this means at least 15 years of age.)
Naturally, this number will change if the age limit is changed, but this is here as a placeholder.
Note: The test is against the birthdate, not the maximum possible birthdate.
Bad data
Note: Original query used the wrong inequality.
SELECT fa.*
, b.b_sex
, b.b_birthdate
, AGE(fa.fa_fol_date, b.b_birthdate) AS age_at_follow
, AGE(fa.fa_fol_date, b.b_birthdate)
- INTERVAL '15 years' AS older_than_limit_by
FROM clean.follow_arrival AS fa
JOIN clean.biography AS b
ON b.b_animid = fa.fa_b_arr_animid
WHERE b.b_sex = 'F'
AND fa.fa_type_of_cycle = 'U'
AND b.b_birthdate
<= fa.fa_fol_date
- INTERVAL '1 year'
- INTERVAL '14 years'
ORDER BY fa.fa_fol_date
, fa.fa_fol_b_focal_animid
, fa.fa_b_arr_animid;
Solution
The number of problem rows was reduced after changing the limit to 14 years from 9 years.
The remaining problems will have to be adjusted in the data. Alternately, we can adjust the hard limit, and set a soft limit of 14 years in the warning system with a note to change the hard limit back once the errors are resolved.
5/7/2026: ICG checked through 2003. When there was a "?" for swelling, I entered '0'. If there was actually a 'U' on the tiki itself, I left "U" in the FA table. Note that a lot of these females are MGF, so the U might be legit.
5/19/2026: MGFs have a dummy birthdate which is why the age is being flagged. The rest are what the observer recorded, so let them in.
UPDATE 2026-07-23: There are 86 failing rows; many but not all are MGF (how to we treat MGF?). Please see below for a summary of this issue and related Problem #47:
8/14/2026 - ICG fixed in Access - all non-MGF entries changed to "0"
UPDATE 2026-08-14: Improved but there are still 48 offending records
note that this summary does not include MFG
| problem | problem_description | age_years | current_cycle | youngest_exact_age | oldest_exact_age | offending_rows | problem_total |
|---|---|---|---|---|---|---|---|
| 47 | U assigned before age 5 | 1 | U | 1 year 4 mons 5 days | 1 year 4 mons 5 days | 1 | 21 |
| 47 | U assigned before age 5 | 2 | U | 2 years 11 mons 26 days | 2 years 11 mons 26 days | 1 | 21 |
| 47 | U assigned before age 5 | 3 | U | 3 years 3 mons 30 days | 3 years 11 mons 1 day | 10 | 21 |
| 47 | U assigned before age 5 | 4 | U | 4 years 5 mons 21 days | 4 years 9 mons 18 days | 9 | 21 |
| 49 | U assigned at age 15 or older | 15 | U | 15 years 1 mon 2 days | 15 years 10 mons 14 days | 16 | 38 |
| 49 | U assigned at age 15 or older | 16 | U | 16 years 19 days | 16 years 11 mons 6 days | 7 | 38 |
| 49 | U assigned at age 15 or older | 17 | U | 17 years 1 mon 19 days | 17 years 1 mon 19 days | 1 | 38 |
| 49 | U assigned at age 15 or older | 18 | U | 18 years 1 mon 18 days | 18 years 8 mons 16 days | 6 | 38 |
| 49 | U assigned at age 15 or older | 32 | U | 32 years 1 mon 21 days | 32 years 5 mons 14 days | 4 | 38 |
| 49 | U assigned at age 15 or older | 33 | U | 33 years 6 days | 33 years 6 days | 4 | 38 |
(#50) There are follow_arrivals where females have a cycle code of n/a
Problem
There are 4,496 follow_arrival rows, for female arriving individuals, that have a sexual cycle code (fa_type_of_cycle) of n/a.
Bad data
select follow_arrival.*
, biography.b_sex
, biography.b_birthdate
from clean.follow_arrival
join clean.biography
on (biography.b_animid = follow_arrival.fa_b_arr_animid)
where biography.b_sex = 'F'
and follow_arrival.fa_type_of_cycle = 'n/a'
order by follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid;
Solution
Change the cycle code to MISS, for these rows. This is done in the clean schema.
This is an indication that future cleanup is required.
(#51) There are follow_arrivals with invalid fa_data_source values
Problem
There are 546 follow_arrival rows that have fa_data_source values that are not one of:
Tiki
Tiki_Mom
Tiki_ID
Brec.
Bad data
Query the easy schema, because the data has been cleaned in the clean schema.
select follow_arrival.*
from easy.follow_arrival
where follow_arrival.fa_data_source <> 'Tiki'
and follow_arrival.fa_data_source <> 'Tiki_Mom'
and follow_arrival.fa_data_source <> 'Tiki_ID'
and follow_arrival.fa_data_source <> 'Brec'
order by follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid;
-- Summarize with:
select follow_arrival.fa_data_source, count(*)
from easy.follow_arrival
where follow_arrival.fa_data_source <> 'Tiki'
and follow_arrival.fa_data_source <> 'Tiki_Mom'
and follow_arrival.fa_data_source <> 'Tiki_ID'
and follow_arrival.fa_data_source <> 'Brec'
group by follow_arrival.fa_data_source
order by follow_arrival.fa_data_source;
Solution
Change BREC and brec to Brec.
Change TikI to Tiki.
Add the other codes:
fa_data_source count Tiki_GM 202 Tiki_PM 165 Tiki_SS 22
Also: Tiki_Mom
(#52) There are follow_arrivals that are almost duplicates
Problem
This entry may end up being multiple problems.
There are follow_arrivals that are near duplicates.
When checking for duplicates on fa_fol_date, fa_fol_b_focal_animid, fa_b_arr_animid
and fa_seq_num, the rows are always unique.
But checking the combination of fa_fol_date, fa_fol_b_focal_animid, fa_b_arr_animid
and fa_time_start,
and fa_time_end yields 184 rows.
Leaving off fa_time_end and just checking the combination of fa_fol_date, fa_fol_b_focal_animid, fa_b_arr_animid
and fa_time_start yields 430 rows.
Why the duplicates?
This entry is a call for a definition of an ARRIVALS row, what does it mean to be a duplicate?
Bad data
-- no duplicates when looking at just sequence number
select fa_fol_date
, fa_fol_b_focal_animid
, fa_b_arr_animid
, fa_seq_num
from clean.follow_arrival
group by fa_fol_date
, fa_fol_b_focal_animid
, fa_b_arr_animid
, fa_seq_num
having count(*) > 1;
-- Checking against start and end time
with dups as
(select fa_fol_date
, fa_fol_b_focal_animid
, fa_b_arr_animid
, fa_time_start
, fa_time_end
from clean.follow_arrival
group by fa_fol_date
, fa_fol_b_focal_animid
, fa_b_arr_animid
, fa_time_start
, fa_time_end
having count(*) > 1)
select *
from clean.follow_arrival
where exists
(select 1
from dups
where follow_arrival.fa_fol_date = dups.fa_fol_date
and follow_arrival.fa_fol_b_focal_animid
= dups.fa_fol_b_focal_animid
and follow_arrival.fa_b_arr_animid = dups.fa_b_arr_animid
and follow_arrival.fa_time_start = dups.fa_time_start
and follow_arrival.fa_time_end = dups.fa_time_end)
order by follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid
, follow_arrival.fa_time_start
, follow_arrival.fa_time_end
, follow_arrival.fa_seq_num;
Solution
IAN FIXED IN ACCESS 8/24/2016
(#53) There are follow_arrivals where the arriving chimp does not exist
Problem
There are 688 84 78 rows, having 3 6 2 different fa_b_arr_animid values, where the fa_b_arr_animid value is not a biography.b_animid value.
Bad Data
-- The rows
select *
from clean.follow_arrival
where not exists
(select 1
from clean.biography
where biography.b_animid = follow_arrival.fa_b_arr_animid)
order by follow_arrival.fa_b_arr_animid
, follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid;
-- A summary of the bad animal ids
select follow_arrival.fa_b_arr_animid, count(*)
from clean.follow_arrival
where not exists
(select 1
from clean.biography
where biography.b_animid = follow_arrival.fa_b_arr_animid)
group by follow_arrival.fa_b_arr_animid
order by follow_arrival.fa_b_arr_animid;
Solution
ICG fixed in MS Access 10/2025. AMA-->AME, OBE-->POR
SRE: still need to address: UWE, GGl, FAC, FAR, GU, and Sl (as of the April 2026 dump)
FIXED UWE, GGl, FAC, FAR, GU, and Sl IN ACCESS 7/3/2026
UPDATE 2026-07-22: still need to resolve FAR (17 records) and FAC (61 records)
UPDATE 2026-08-18: ICG fixed in Access. FAR-->FAY, FAC-->FIC
(#54) There are follow_arrivals where females that are too young have a cycle state code that is not 0, U, or MISS
Problem
There are 26 42 follow_arrival rows, for female arriving individuals, that have a sexual cycle code indicating sexual swelling but are less than 8 years of age.
Note: The test is against the birthdate, not the minimum possible birthdate.
Bad data
select follow_arrival.*, biography.b_sex, biography.b_birthdate
from clean.follow_arrival
join clean.biography
on (biography.b_animid = follow_arrival.fa_b_arr_animid)
where biography.b_sex = 'F'
and follow_arrival.fa_type_of_cycle <> '0'
and follow_arrival.fa_type_of_cycle <> 'U'
and follow_arrival.fa_type_of_cycle <> 'MISS'
and biography.b_birthdate
> (follow_arrival.fa_fol_date
- '8 years'::INTERVAL
)
and follow_arrival.fa_b_arr_animid <> 'MGF'
and follow_arrival.fa_b_arr_animid <> 'MGF2'
and follow_arrival.fa_b_arr_animid <> 'MGF3'
order by follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid;
Solution
IAN TO FIX - CHANGE ALL TO "U"
That works. Would it be better to leave the original MS Access data untouched and have the conversion process make the change in the clean schema?
UPDATE 2026-07-23: The proposed solution has been applied to the clean schema (see commit 06c4178). However, please note that this does not solve Problem #47 and, in fact, forces the fa_type_of_cycle type of records associated with Problem #47 to 'U'.
(#55) There are community_membership rows that place an individual in a community before birth
Problem
There is one individual who is placed in a community, once, before birth.
Note: The test is against the birthdate, not the minimum possible birthdate.
Bad data
select b.b_animid, b.b_birthdate, cm.cm_start_date
from clean.community_membership as cm
join clean.biography as b
on (b.b_animid = cm.cm_b_animid)
where cm.cm_start_date < b.b_birthdate
order by b.b_animid, cm.cm_start_date;
Bad solution
ICG fixed FN community starte date in MS Access - oct 2025
(#56) There are follow_arrival focal animids with trailing spaces
Problem
There are 144 follow arrivals where the focal id has trailing spaces.
Bad data
select * from follow_arrival where fa_fol_b_focal_animid <> rtrim(fa_fol_b_focal_animid);
Solution
Remove the trailing spaces in table in the clean schema.
(#57) GROOM_BOUT duplicate keys
Problem
The data dump says that the GROOM_BOUT table has a primary key consisting of, in order, the columns: GRM_FOL_date, GRM_FOL_B_focal_AnimId, GRM_time_begin, GRM_B_partner_AnimId
But these columns contain duplicate values.
Bad Data
The duplicate values can be listed (from the raw schema) with:
SELECT *
FROM raw."GROOM_BOUT"
JOIN (SELECT "GRM_FOL_date" AS the_date
, "GRM_FOL_B_focal_AnimId" AS the_animid
, "GRM_time_begin" AS the_time
, "GRM_B_partner_AnimId" AS the_partner
FROM raw."GROOM_BOUT"
GROUP BY "GRM_FOL_date"
, "GRM_FOL_B_focal_AnimId"
, "GRM_time_begin"
, "GRM_B_partner_AnimId"
HAVING count(*) > 1
) AS gb
ON ("GROOM_BOUT"."GRM_FOL_date" = gb.the_date
AND "GROOM_BOUT"."GRM_FOL_B_focal_AnimId" = gb.the_animid
AND "GROOM_BOUT"."GRM_time_begin" = gb.the_time
AND "GROOM_BOUT"."GRM_B_partner_AnimId" = gb.the_partner);
SOLUTION
ICG FIXED IN ACCESS - ALL WERE LEGIT DUPLICATES. REMOVED. 8/18/2026
(#58) OTHER_SPECIES duplicate keys
Problem
The data dump says that the OTHER_SPECIES table has a primary key consisting of, in order, the columns: OS_FOL_date, OS_FOL_B_focal_AnimID, OS_time_begin
But these columns contain duplicate values.
Bad Data
The duplicate values can be listed (from the raw schema) with:
SELECT *
FROM raw."OTHER_SPECIES"
JOIN (SELECT "OS_FOL_date" AS the_date
, "OS_FOL_B_focal_AnimID" AS the_animid
, "OS_time_begin" AS the_time
FROM raw."OTHER_SPECIES"
GROUP BY "OS_FOL_date"
, "OS_FOL_B_focal_AnimID"
, "OS_time_begin"
HAVING count(*) > 1
) AS os
ON ("OTHER_SPECIES"."OS_FOL_date" = os.the_date
AND "OTHER_SPECIES"."OS_FOL_B_focal_AnimID" = os.the_animid
AND "OTHER_SPECIES"."OS_time_begin" = os.the_time);
Note
ICG fixed in Access 8/18/2026
(#59) Zero BIOGRAPHY.b_animid_num values
Problem
SokewDB requires that the animal ID number be greater than 0, or NULL, but some (what seem to be rows for babys) have a 0 value.
Bad Data
The 0 values can be listed (from the clean schema) with:
select * from clean.biography where b_animid_num = 0;
Solution
Change the 0 values to NULL.
This is a brute-force, but adequate, solution because it does not validate anything concerning the rows affected.
(#60) Invalid biography_update_log.made_by values
Problem
There is a biography_update_log.made_by value (SF/EVL) that is not a person. (Not on the PEOPLE table.)
Bad Data
The invalid values can be listed (from the clean schema) with:
SELECT *
FROM clean.biography_update_log
WHERE made_by IS NOT NULL
AND NOT EXISTS (SELECT 1
FROM clean.people
WHERE people.person = biography_update_log.made_by);
SOLUTION
IAN CHANGED THE SINGLE SF/EVL ENTRY TO EVL IN ACCESS 3/25/2026
update: this fix was not reflected in the 2026-07-12 dump.
Codified in clean per commit b793345f4098fa012d559b8f2d01f06d5418126c.
-- Problem #60 BIOGRAPHY_UPDATE_LOG.MadeBy contains the invalid combination -- person code SF/EVL. The source-data resolution is to use EVL. UPDATE biography_update_log SET made_by = 'EVL' WHERE made_by = 'SF/EVL';
* (#61) Invalid follow date/focals pairs in follow_arrival
Problem
There are follow_arrival.fa_fol_b_animid and follow_arrival.fa_fol_date value combinations that do not exist in follow.
Bad Data
The invalid values can be listed (from the clean schema) with:
SELECT *
FROM clean.follow_arrival
WHERE NOT EXISTS
(SELECT 1
FROM clean.follow
WHERE follow.fol_date = follow_arrival.fa_fol_date
AND follow.fol_b_animid
= follow_arrival.fa_fol_b_focal_animid)
ORDER BY follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid
, follow_arrival.fa_time_start
, follow_arrival.fa_time_end;
SOLUTION
These are all cases when a juvenile arrival has been extracted from b record notes, but there seems to be no tiki. Do we add WATCHES for these?
(#62) There are follow_arrival rows with arriving animids with trailing spaces
Problem
There are 39 follow arrivals where the arriving animid has trailing spaces.
Bad data
select * from easy.follow_arrival where fa_b_arr_animid <> rtrim(fa_b_arr_animid);
Solution
Remove the trailing spaces in table in the clean schema. Addressed 9c114ac.
(#63) There are follow_arrival rows with NULL fa_data_source values
Problem
There are 5,020 follow arrivals where the fa_data_source is NULL.
Bad data
SELECT *
FROM easy.follow_arrival
WHERE fa_data_source IS NULL
ORDER BY follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid
, follow_arrival.fa_time_start
, follow_arrival.fa_time_end;
Solution
KARL TO FIX
Create a none value in ARRIVAL_SOURCES, and use that value instead of NULL in follow_arrival table in the clean schema. Addressed 8c05232.
(#64) There are follow_arrival rows with NULL fa_type_of_certainty values
Problem
There is 1 follow arrivals where the fa_type_of_certainty is NULL.
Bad data
SELECT *
FROM easy.follow_arrival
WHERE fa_type_of_certainty IS NULL
ORDER BY follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid
, follow_arrival.fa_time_start
, follow_arrival.fa_time_end;
Solution
IAN fixed in Access 3/25/2026
(#65) There are AGGRESSION_EVENT rows that have "YES" or a space as a ae_bad_observeration_flag value
Problem
There are 3,079 rows where the ae_bad_observation_flag is a space, and 1 row where it is YES.
Bad data
SELECT COALESCE(ae.ae_bad_observation_flag, 'NULL'), count(*) FROM easy.aggression_event AS ae GROUP BY ae.ae_bad_observation_flag;
Solution
KARL: Treat spaces (along with NULL) as FALSE, and YES (along with X() as TRUE.
Stevan: Change the data in the clean schema to contain only boolean values, based on the above.
addressed as part of the solution to problem #74 (commit efa75af037a5e22b2b43675b435f472193a6672f)
(#66) There are AGGRESSION_EVENT rows that have a space as a ae_decided_flag value
Problem
There are 2,041 rows where the ae_decided_flag is a space.
Bad data
SELECT COALESCE(ae.ae_decided_flag, 'NULL'), count(*) FROM easy.aggression_event AS ae GROUP BY ae.ae_decided_flag;
Solution
KARL: Treat spaces (along with NULL) as FALSE.
Stevan: Fix in the clean schema.
addressed as part of the solution to problem #74 (commit efa75af037a5e22b2b43675b435f472193a6672f)
(#67) There are AGGRESSION_EVENT rows that have a space or an x as a ae_multiple_aggressor_flag value
Problem
There are 2,397 rows where the ae_multiple_aggressor_flag is a space and 2 rows where the value is an x.
Bad data
SELECT COALESCE(ae.ae_multiple_aggressor_flag, 'NULL'), count(*) FROM easy.aggression_event AS ae GROUP BY ae.ae_multiple_aggressor_flag;
Solution
Treat spaces (along with NULL) as FALSE, and x (along with X) as TRUE.
addressed as part of the solution to problem #74 (commit efa75af037a5e22b2b43675b435f472193a6672f)
(#68) There are AGGRESSION_EVENT rows that have a space as a ae_multiple_recipient_flag value
Problem
There are 2,528 rows where the ae_multiple_recipient_flag is a space.
Bad data
SELECT COALESCE(ae.ae_multiple_recipient_flag, 'NULL'), count(*) FROM easy.aggression_event AS ae GROUP BY ae.ae_multiple_recipient_flag;
Solution
Treat spaces (along with NULL) as FALSE.
addressed as part of the solution to problem #74 (commit efa75af037a5e22b2b43675b435f472193a6672f)
(#69) There are duplicate year*community records in AGGRESSION_EVENT_LOG
Problem
There are 11 duplicate pairs of year*community records in AGGRESSION_EVENT_LOG. There can be, at most, one row per-community, per-year. The duplicates each have a `b_rec_english` value of `ALL` or `F-F (ALL); F-M (ALL); M-M (ALL)`.
Bad data
SELECT *
FROM clean.aggression_event_log
JOIN (
SELECT
year,
community
FROM
clean.aggression_event_log
GROUP BY
year,
community
HAVING count(*) > 1
) AS dups ON (
dups.year = aggression_event_log.year
AND dups.community = aggression_event_log.community
)
order by
aggression_event_log.year,
aggression_event_log.community ;
Solution
IAN fixed 8/18/2026
(#70) There are AGGRESSION_EVENT rows that do not have a matching FOLLOW for the same focal ID and date.
Problem
There are 1494 records where clean.aggression_event does not have a matching row in clean.follow for the same focal ID and date.
Bad data
WITH problem_rows AS (
SELECT
ae.*,
EXISTS (
SELECT 1
FROM clean.follow f_id
WHERE f_id.fol_b_animid = ae.ae_fol_b_focal_id
) AS focal_id_exists_in_follow
FROM clean.aggression_event ae
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
)
SELECT
pr.ae_date,
pr.ae_time,
pr.ae_fol_b_focal_id,
pr.ae_b_aggressor_id,
pr.ae_b_recipient_id,
pr.ae_source,
pr.ae_full_description,
CASE
WHEN pr.focal_id_exists_in_follow THEN 'missing_follow_on_same_date'
ELSE 'focal_id_not_found_in_follow'
END AS issue_type
FROM problem_rows pr
ORDER BY
pr.ae_date,
pr.ae_fol_b_focal_id,
pr.ae_time;
Resolution
Resolved by commit `1cc9e694a1f6a53c396d9074f77d5d279018721b`.
A matching FOLLOW row is not required for an aggression event. WATCHES represents the observation context independently of FOLLOW. A B-record WATCHES row may represent either a follow or an ad-hoc observation such as an aggression.
During conversion, `load_aggressions.m4` looks for an existing B-record WATCHES row having the aggression event's focal individual and date. If no such row exists, the loader creates one using the aggression event's focal, community, and date. The aggression's EVENTS row is then related to that WATCHES row.
The original diagnostic query will continue to identify aggression records without matching FOLLOW rows. These rows are not necessarily bad data and are not expected to disappear.
* (#71) There are AGGRESSION_EVENT animal ids that are not reflected in among FOLLOW animal ids.
Problem
There are 16 unique(!) aggression_event.ae_fol_b_focal_id records that are not reflected among follow.fol_b_animid values.
Processing note
All of the problematic records are reflected in the rows excluded as part of Problem #70. As such, there is not a separate exclusion clause in the conversion process for these data. As a corollary, addressing all issues in Problem #70 would address all Problem #71 infractions well.
Problem #70 concerns aggression rows without a matching follow for the same focal ID and date. Canonical master solves that relationship problem by creating a B-record WATCHES row when no suitable watch exists. An aggression therefore does not inherently require a FOLLOW.
Problem #71 is a stricter subset: its focal IDs never occur among follow focal IDs. More importantly, a read-only check of the local data found that the documented 16 IDs are also absent from clean.biography, including values such as DL DUK, GA GGL, GROUP, Males, and stranger.
WATCHES.AnimID → BIOGRAPHY_DATA.AnimID
Consequently:
- If a #71 focal ID is a valid biography ID, the #70 WATCHES solution handles it. - If it is not a valid biography ID, creating the WATCHES row fails its foreign key. - The documented 16 IDs appear to be in the latter category and therefore require a separate source-data decision or clean-stage mapping.
There is also a branch-specific complication: the temporary branch loader still has the old #70 WHERE EXISTS filter. It excludes all rows lacking a matching follow, including every #71 row. Thus the successful mung conversion does not demonstrate that these rows were converted. Canonical master does not have that filter and would expose invalid #71 focal IDs when attempting to create their WATCHES rows.
Therefore, the Problem #71 processing note is too broad. The #70 solution removes the requirement for a corresponding follow, but it does not resolve invalid or composite focal animal IDs. Problem #71 should remain a separate unresolved data-quality problem, and mung’s stale #70 exclusion should eventually be removed when testing the full solution.
Bad data
SELECT
ae_fol_b_focal_id,
COUNT(*) AS row_count
FROM clean.aggression_event
WHERE ae_fol_b_focal_id NOT IN (
SELECT DISTINCT fol_b_animid
FROM clean.follow
WHERE fol_b_animid IS NOT NULL
)
GROUP BY ae_fol_b_focal_id
ORDER BY
row_count DESC,
ae_fol_b_focal_id
;
SOLUTION
IAN FIXED IN ACCESS 8/25/2026
(#72) There are AGGRESSION_EVENT recipient certainty flags other than `Y` or `N` (required).
Problem
There are 28,322 AGGRESSION_EVENT rows that have a ae_recipient_certainty_flag other than `Y` or `N` as required.
Bad data
Full record
SELECT
ae.*,
COALESCE(ae_recipient_certainty_flag, '<NULL>') AS raw_flag
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND UPPER(BTRIM(COALESCE(ae.ae_recipient_certainty_flag, ''))) NOT IN ('N', 'Y') ;
Summary of non-compliant values
SELECT
COALESCE(ae_recipient_certainty_flag, '<NULL>') AS raw_flag,
COUNT(*) AS row_count
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND UPPER(BTRIM(COALESCE(ae.ae_recipient_certainty_flag, ''))) NOT IN ('N', 'Y')
GROUP BY COALESCE(ae_recipient_certainty_flag, '<NULL>')
ORDER BY row_count DESC, raw_flag;
Solution
Ian or Elizabeth to please confirm.
- sokwe conversions:
- ' ' to '0'
- 'N' to '0'
- 'NO' to '0'
- 'X' to '1'
- 'Y%' to '1'
ICG AND ELO CONFIRM 7/3/2026
KARL
Stevan: The usual. Fix in the clean schema.
addressed as part of the solution to problem #74 (commit efa75af037a5e22b2b43675b435f472193a6672f)
(#73) There are AGGRESSION_EVENT aggression_event.ae_b_recipient_id values that are NULL.
Problem
There are 5,820 AGGRESSION_EVENT rows for which the aggression_event.ae_b_recipient_id is NULL.
Bad data
SELECT
ae.ae_date,
ae.ae_time,
ae.ae_fol_b_focal_id,
ae.ae_b_aggressor_id,
ae.ae_b_recipient_id,
ae.ae_recipient_certainty_flag,
ae.ae_full_description,
ae.ae_source,
ae.ae_comments
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND ae.ae_b_recipient_id IS NULL
ORDER BY ae.ae_date, ae.ae_fol_b_focal_id, ae.ae_time;
SOLUTION
CAN THESE ALL BE NULL? IF NOT, THEN WE NEED TO DO SOMETHING ABOUT INSTANCES WHEN 'GROUP' IS THE TARGET.
Karl's comment
The usual solution is to have the unknown individual, UNK, be the target. If NULL has some other special meaning, other than we don't know at whom the aggression was directed, we could make up another "special" BIOGRAPHY_DATA row, similar to the UNK, MGF, MGM, etc. rows.
Yes, use UNK.
(#74) There are AGGRESSION_EVENT event flags (multiple columns) that have values other than X or NULL.
Problem
Among AGGRESSION_EVENT flags (ae_bad_observation_flag, ae_bristle_flag, ae_chase_flag, ae_contact_flag, ae_contact_flag, ae_decided_flag, ae_display_flag, ae_multiple_aggressor_flag, ae_multiple_recipient_flag, ae_vocal_flag, ae_vocal_flag) there are 2,864 rows that have a value other than `X` or NULL (required) for one or more of the flags.
Bad data
Full record
SELECT
ae.ae_date,
ae.ae_time,
ae.ae_fol_b_focal_id,
ae.ae_b_aggressor_id,
ae.ae_b_recipient_id,
ae.ae_decided_flag,
ae.ae_multiple_aggressor_flag,
ae.ae_multiple_recipient_flag,
ae.ae_bad_observation_flag,
ae.ae_bristle_flag,
ae.ae_display_flag,
ae.ae_chase_flag,
ae.ae_contact_flag,
ae.ae_vocal_flag,
ae.ae_source,
ae.ae_comments
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND ae.ae_b_recipient_id IS NOT NULL
AND (
(ae.ae_decided_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_decided_flag)) <> 'X')
OR (ae.ae_multiple_aggressor_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_multiple_aggressor_flag)) <> 'X')
OR (ae.ae_multiple_recipient_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_multiple_recipient_flag)) <> 'X')
OR (ae.ae_bad_observation_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_bad_observation_flag)) <> 'X')
OR (ae.ae_bristle_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_bristle_flag)) <> 'X')
OR (ae.ae_display_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_display_flag)) <> 'X')
OR (ae.ae_chase_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_chase_flag)) <> 'X')
OR (ae.ae_contact_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_contact_flag)) <> 'X')
OR (ae.ae_vocal_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_vocal_flag)) <> 'X')
)
ORDER BY ae.ae_date, ae.ae_fol_b_focal_id, ae.ae_time;
Summary of non-compliant values
WITH base AS (
SELECT ae.*
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND ae.ae_b_recipient_id IS NOT NULL
),
flag_values AS (
SELECT 'ae_decided_flag' AS flag_name, COALESCE(ae_decided_flag, '<NULL>') AS raw_value FROM base
UNION ALL
SELECT 'ae_multiple_aggressor_flag', COALESCE(ae_multiple_aggressor_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_multiple_recipient_flag', COALESCE(ae_multiple_recipient_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_bad_observation_flag', COALESCE(ae_bad_observation_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_bristle_flag', COALESCE(ae_bristle_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_display_flag', COALESCE(ae_display_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_chase_flag', COALESCE(ae_chase_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_contact_flag', COALESCE(ae_contact_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_vocal_flag', COALESCE(ae_vocal_flag, '<NULL>') FROM base
)
SELECT
flag_name,
raw_value,
COUNT(*) AS row_count
FROM flag_values
WHERE raw_value <> '<NULL>'
AND UPPER(BTRIM(raw_value)) <> 'X'
GROUP BY flag_name, raw_value
ORDER BY flag_name, row_count DESC, raw_value;
Solution
Ian or Elizabeth to please confirm.
- clean schema conversions (mimic patterns in Access):
- `?` to ' '
- `Y%` to `X`
- `X%` to `X`
- sokwe conversions:
- ' ' to '0'
- 'X' to '1'
ICG AND ELO CONFIRM 7/3/26
addressed with commits efa75af037a5e22b2b43675b435f472193a6672f and 921b5ca3e9ca3d1e06fe189b4775aae7dcc484bf note that the solution to problem #74 was an umbrella fix that also addressed #65-68, and #72
* (#75) There are AGGRESSION_EVENT records where ae_extracted_by values are not in the people table.
Problem
There are 28,513 AGGRESSION_EVENT records where ae_extracted_by does not match a person in the PEOPLE table.
Bad data
Full record
WITH scoped AS (
SELECT ae.*
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND ae.ae_b_recipient_id IS NOT NULL
)
SELECT
s.ae_date,
s.ae_time,
s.ae_fol_b_focal_id,
s.ae_b_aggressor_id,
s.ae_b_recipient_id,
s.ae_extracted_by,
s.ae_source,
s.ae_full_description,
s.ae_comments
FROM scoped s
WHERE NOT EXISTS (
SELECT 1
FROM people p
WHERE p.person = BTRIM(COALESCE(s.ae_extracted_by, ''))
OR LOWER(p.name) = LOWER(BTRIM(COALESCE(s.ae_extracted_by, '')))
)
ORDER BY s.ae_date, s.ae_fol_b_focal_id, s.ae_time;
Summary of non-compliant values
WITH scoped AS (
SELECT ae.*
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND ae.ae_b_recipient_id IS NOT NULL
)
SELECT
BTRIM(COALESCE(ae_extracted_by, '')) AS raw_extractedby,
COUNT(*) AS row_count
FROM scoped s
WHERE NOT EXISTS (
SELECT 1
FROM people p
WHERE p.person = BTRIM(COALESCE(s.ae_extracted_by, ''))
OR LOWER(p.name) = LOWER(BTRIM(COALESCE(s.ae_extracted_by, '')))
)
GROUP BY BTRIM(COALESCE(ae_extracted_by, ''))
ORDER BY row_count DESC, raw_extractedby;
SOLUTION
ALLOW FOR NOW. NOT SURE IF/HOW THE PEOPLE TABLE IS POPULATED.
Karl's comment
We populate the PEOPLE however we like. We just make things up.
We can add all these values to the table as different people.
However, unless you set the PEOPLE.Active to FALSE
you run the risk of future data entry using a bunch of odd or outdated people.
(#76) There are AGGRESSION_EVENT rows where the Actor/Actee are not in the biography table.
Problem
There are 10,866 combined (but see following!) rows where the ae_b_aggressor_id and/or ae_b_recipient_id is not in the BIOGRAPHY table.
Notes:
- The row count is inflated by a CROSS LATERAL JOIN, which yields the number of combined, pivoted records for both ae_b_aggressor_id and ae_b_recipient, not the actual number of confounding AGGRESSION_EVENT rows.
- The queries currently exclude ae_b_recipient_id values that are NULL, which seems a related but separte problem; the number of rows jumps to 10,357 if ae_b_recipient_id = NULL are included.
- How are we treating ae_b_recipient_id values such as `group`, `males`, `females`, etc.?
- Whitespace is considered in the incongruence such that, for example, `VIN ` (with a space) does not match `VIN` in biography.
Bad data
Full record
SELECT
ae.ae_date,
ae.ae_time,
ae.ae_fol_b_focal_id,
v.role_name,
v.participant,
ae.ae_b_aggressor_id,
ae.ae_b_recipient_id,
ae.ae_source,
ae.ae_full_description,
ae.ae_comments
FROM clean.aggression_event ae
CROSS JOIN LATERAL (
VALUES
('Actor'::text, ae.ae_b_aggressor_id),
('Actee'::text, ae.ae_b_recipient_id)
) AS v(role_name, participant)
WHERE v.participant IS NOT NULL
AND v.participant <> ''
AND NOT EXISTS (
SELECT 1
FROM clean.biography b
WHERE b.b_animid = v.participant
)
ORDER BY ae.ae_date, ae.ae_fol_b_focal_id, ae.ae_time, v.role_name;
Summary of non-compliant values
trimmed_match denotes a match between Actor/Actee and biography if whitespace is trimmed.
SELECT
v.role_name,
v.participant AS raw_participant,
b_trimmed.b_animid AS trimmed_match,
COUNT(*) AS row_count
FROM clean.aggression_event ae
CROSS JOIN LATERAL (
VALUES
('Actor'::text, ae.ae_b_aggressor_id),
('Actee'::text, ae.ae_b_recipient_id)
) AS v(role_name, participant)
LEFT JOIN clean.biography b_trimmed
ON BTRIM(v.participant) = BTRIM(b_trimmed.b_animid)
WHERE v.participant IS NOT NULL
AND v.participant <> ''
AND NOT EXISTS (
SELECT 1
FROM clean.biography b
WHERE b.b_animid = v.participant
)
GROUP BY v.role_name, v.participant, b_trimmed.b_animid
ORDER BY row_count DESC, v.role_name, v.participant;
SOLUTION
ICG fixed in Access 7/7/2026. replaced "group", "males", etc with UNK. Fixed bad IDs. Updated comments field to reflect original entry terms.
Karl comment
You can't just "allow them". You can make rows in BIOGRAPHY_DATA that have AnimID values that correspond to the values that appear. It's hard to say whether or not these rows would have to be "special". Probably not, because anybody can aggress against anybody. But if you're going to make a whole bunch of "dummy" individuals (138 of them, to be exact), it might be prudent to add a flag to BIOGRAPHY_DATA that says whether or not the row corresponds to a specific chimpanzee. (That flag could even be used to filter the dummy individuals out of the BIOGRAPHY view.)
(#77) There are AGGRESSION_EVENT rows where ae_recipient_behavior is NULL or an empty string.
Problem
There are 9,102 records where ae_recipient_behavior is NULL or an empty string.
Bad data
SELECT * FROM clean.aggression_event WHERE ae_recipient_behavior IS NULL OR ae_recipient_behavior = '' ;
SOLUTION
ALLOW FOR NOW
Karl's comments
Change the NULL values to the empty string in the clean schema.
Addressed with commit a84462b47c8b2100caa02958623817981179d068.
(#78) There are AGGRESSION_EVENT rows where ae_full_description is NULL or an empty string.
Problem
There are 4,718 records where ae_full_description is NULL or an empty string.
Bad data
SELECT * FROM clean.aggression_event WHERE ae_full_description IS NULL OR ae_full_description = '' ;
SOLUTION
ALLOW FOR NOW.
Karl's comments
Change the NULL values to the empty string in the clean schema. Addressed with commit a84462b47c8b2100caa02958623817981179d068.
(#79) There are AGGRESSION_EVENT participants outside their valid study participation window in the biography data.
Problem
There are 46 6 records where an Actor and/or Actee are in the AGGRESSION_EVENT table but outside their valid study participation window in the biography data. Note that the query strips white space when comparing the ids of aggression participant ids and biography animid ids given that the focus here is on checking for events outside of defined date ranges rather than matching ids and assuming that issues concerning white space will be resolved (also in the migration work-around). The query further winnows aggression events for which there is a valid follow record (not in the migration work-around).
Bad data
WITH candidate_agg AS (
SELECT
ae.ae_date,
ae.ae_time,
ae.ae_fol_b_focal_id,
BTRIM(COALESCE(ae.ae_b_aggressor_id, '')) AS actor_id,
BTRIM(COALESCE(ae.ae_b_recipient_id, '')) AS actee_id,
ae.ae_source,
ae.ae_full_description
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND ae.ae_b_recipient_id IS NOT NULL
AND EXISTS (
SELECT 1
FROM clean.biography b
WHERE b.b_animid = BTRIM(COALESCE(ae.ae_b_aggressor_id, ''))
)
AND EXISTS (
SELECT 1
FROM clean.biography b
WHERE b.b_animid = BTRIM(COALESCE(ae.ae_b_recipient_id, ''))
)
),
participants AS (
SELECT
c.ae_date,
c.ae_time,
c.ae_fol_b_focal_id,
'Actor' AS role,
c.actor_id AS participant,
c.ae_source,
c.ae_full_description
FROM candidate_agg c
UNION ALL
SELECT
c.ae_date,
c.ae_time,
c.ae_fol_b_focal_id,
'Actee' AS role,
c.actee_id AS participant,
c.ae_source,
c.ae_full_description
FROM candidate_agg c
)
SELECT
p.ae_date,
p.ae_time,
p.ae_fol_b_focal_id,
p.role,
p.participant,
b.b_entrydate,
b.b_departdate,
CASE
WHEN p.ae_date < b.b_entrydate THEN 'before_entry'
WHEN p.ae_date > b.b_departdate THEN 'after_departure'
ELSE 'ok'
END AS violation_type,
p.ae_source,
p.ae_full_description
FROM participants p
JOIN clean.biography b
ON b.b_animid = p.participant
WHERE p.ae_date < b.b_entrydate
OR p.ae_date > b.b_departdate
ORDER BY p.ae_date, p.ae_fol_b_focal_id, p.ae_time, p.role, p.participant;
SOLUTION
ICG FIXED IN ACCESS 7/6/2026
Karl comment
There is no good way to "allow for now".
We could change the under study dates (EntryDate, DepartDate) of the affected individuals "for now". That would "solve" the problem.
Or just don't convert the data and let the users get the info from the clean schema and integrate it into their processing.
Update 2026-07-26: row count is much improved but there are six records that violate this constraint.
(#80) There are AGGRESSION_EVENT rows where ae_aggressor_behavior is NULL or an empty string.
Problem
There are 143 records where ae_aggressor_behavior is NULL or an empty string.
Bad data
SELECT * FROM clean.aggression_event WHERE ae_aggressor_behavior IS NULL OR BTRIM(ae_aggressor_behavior) = '' ORDER BY ae_date, ae_fol_b_focal_id, ae_time;
SOLUTION
ALLOW FOR NOW. EVENTUALLY A VOLUNTEER CAN POPULATE USING THE FULL DESCRIPTION FIELD.
Change the NULL values to the empty string in the clean schema.
Addressed with commit a84462b47c8b2100caa02958623817981179d068.
(#81) There are AGGRESSION_EVENT rows where ae_time is NULL or outside the allowable window ('04:00:00', '20:00:00').
Problem
There are 112 AGGRESSION_EVENT rows where ae_time is NULL or outside the allowable window ('04:00:00', '20:00:00').
Bad data
SELECT
ae_date,
ae_time,
ae_fol_b_focal_id,
ae_b_aggressor_id,
ae_b_recipient_id,
ae_source,
ae_full_description,
CASE
WHEN ae_time IS NULL THEN 'null_time'
WHEN ae_time < '04:00:00'::time THEN 'below_min'
WHEN ae_time > '20:00:00'::time THEN 'above_max'
END AS time_issue
FROM clean.aggression_event
WHERE ae_time IS NULL
OR ae_time < '04:00:00'::time
OR ae_time > '20:00:00'::time
ORDER BY ae_date, ae_fol_b_focal_id, ae_time;
SOLUTION
MOST ARE NULL VALUES BECAUSE THEY WERE 'BOOK EXTRACTS'. ASSIGN A 'DUMMY' TIME?
ICG FIXED THE FEW THAT WERE NOT BETWEEN 4:00 AND 20:00 IN ACCESS 7/6/2026
Karl's comment
Allowing means changing the "too early" and "too late" limits for all times, appearing anywhere in the db. This is easy, but opens the door for other data errors.
The change to the time limits cannnot (easily) be restricted to just the table that is converted. Well.... It can. But the change will not be reflected in the documentation. (And it'll be completely annoying because a change will have to be re-made each time the conversion program is re-run, which is a lot because we have a lot of errors to resolve.)
Changing the limits back is less than easy. A separate SQL statement needs to be run for each time column that appears in the database. Probably better is to dump the database content, rebuild the database tables, etc., and then re-load the database content. If any time values are out of bounds, the re-load of the database content will fail.
Executive Summary: Anything's possible but you're coloring outside the lines here.
resolution
The schema already has a deliberate non-NULL representation for an unknown aggression time: 00:00, defined as sdb_no_time.
The design is explicit:
- EVENTS.Start and EVENTS.Stop remain NOT NULL. - Normal event times must be between 04:00 and 20:00. - Aggressions alone may use 00:00 to mean “no time was recorded.” - Both start and stop must be 00:00 together.
Existing implementation Commit 37bb5493b7571084e6d57aa45c5440dcb0db357e Karl O. Pinc 2026-07-15
(#82) There are AGGRESSION_EVENT rows where severity (ae_fight_category) is not an allowable value.
Problem
There are 2,041 AGGRESSION_EVENT rows where severity (ae_fight_category) is not an allowable value [('unrated', '0', '1', '2', '?')]. Note that all non-compliant values are NULL or some variation of white space.
Bad data
Full record
SELECT
ae_date,
ae_time,
ae_fol_b_focal_id,
ae_b_aggressor_id,
ae_b_recipient_id,
ae_fight_category,
BTRIM(COALESCE(ae_fight_category, '')) AS normalized_fight_category,
ae_source,
ae_full_description,
ae_comments
FROM clean.aggression_event
WHERE ae_fight_category IS NOT NULL
AND (
BTRIM(ae_fight_category) = ''
OR BTRIM(ae_fight_category) NOT IN ('unrated', '0', '1', '2', '?')
)
ORDER BY ae_date, ae_fol_b_focal_id, ae_time;
Summary of non-compliant values
SELECT
BTRIM(COALESCE(ae_fight_category, '')) AS offending_value,
COUNT(*) AS row_count
FROM clean.aggression_event
WHERE ae_fight_category IS NOT NULL
AND (
BTRIM(ae_fight_category) = ''
OR BTRIM(ae_fight_category) NOT IN ('unrated', '0', '1', '2', '?')
)
GROUP BY BTRIM(COALESCE(ae_fight_category, ''))
ORDER BY row_count DESC, offending_value;
Fine-grain assessment of non-compliant values
Note the white space problem.
select distinct '"' || ae_fight_category || '"' from clean.aggression_event ;
SOLUTION
ALLOW FOR NOW - MOST DATA ENTERERS DIDN'T FILL THIS COLUMN IN
Karl's comment
There seems to be only a problem because NULL values exist. The other non-conformant data seems to be gone from MS Access. So make a fight category code that is "no data" and clean up the data in the clean schema.
NULL means unrated
resolution
Added `NODATA` to `AGG_SEVERITIES` for aggression records where no fight-category data were recorded. This is distinct from `unrated`, which means the aggression was not evaluated for severity.
During construction of the `clean` schema, `ae_fight_category` values are trimmed. `NULL`, empty-string, and whitespace-only values are converted to `NODATA`. The aggression loader then copies the cleaned value directly into `AGGRESSIONS.Severity`, allowing unexpected nonblank codes to remain visible as conversion errors.
Implemented in commit `d911ef2` (`Resolve Problem #82 with NODATA severity`). A complete conversion using the updated code succeeded.
* (#83) There are AGGRESSION_EVENT rows where the Actor and Actee have the same animal ID.
Problem
There are 1,832 363 AGGRESSION_EVENT rows where `ae_b_aggressor_id` and `ae_b_recipient_id` share the same animal id.
Bad data
WITH potential_role_dupes AS (
SELECT
ae.ae_date,
ae.ae_time,
ae.ae_fol_b_focal_id,
ae.ae_b_aggressor_id,
ae.ae_b_recipient_id,
ae.ae_source,
ae.ae_full_description,
ae.ae_comments,
ae.dup
FROM clean.aggression_event ae
WHERE ae.ae_b_recipient_id IS NOT NULL
AND ae.ae_b_aggressor_id IS NOT NULL
)
SELECT
ae_date,
ae_time,
ae_fol_b_focal_id,
ae_b_aggressor_id,
ae_b_recipient_id,
ae_source,
ae_full_description,
ae_comments,
dup
FROM potential_role_dupes
WHERE ae_b_aggressor_id = ae_b_recipient_id
ORDER BY ae_date, ae_fol_b_focal_id, ae_time;
Processing note
This data problem generates the following error but note that, in practice, this error could be generated for other reasons as well.
./load_chunks.sh load_aggressions.m4 clean.aggression_event
psql:<stdin>:394: ERROR: duplicate key value violates unique constraint "On ROLES, Participant + EID must be unique"
DETAIL: Key (participant, eid)=(FD, 759513) already exists.
CONTEXT: SQL statement "INSERT INTO roles (
eid
, role
, participant)
VALUES (
CURRVAL('events_eid_seq')
, 'Actee'
, this_ae.ae_b_recipient_id)"
PL/pgSQL function inline_code_block line 186 at SQL statement
make: *** [Makefile:373: load_aggressions] Error 3
SOLUTION
ALLOW FOR NOW. EVENTUALLY A VOLUNTEER CAN CHECK AND FIX
Karl's comment
If we allow, we allow for all event types, not just aggressions. Groomings, matings, etc would all allow "self-dealing". I'd rather not.
How about not converting this data and letting people who want it get it from the clean schema?
do not convert these rows, fix them later
(#84) Some follows have a community with trailing spaces
Problem
There are 3 follows where the fol_cl_community_id has trailing spaces.
Bad Data
Cleaned in clean schema, so query easy schema.
select '"' || fol_cl_community_id || '"' AS fol_cl_community_id_untrimmed, * FROM easy.follow WHERE RTRIM(fol_cl_community_id) <> fol_cl_community_id;
SOLUTION
KARL TO FIX
Fix by trimming spaces in the clean schema.
(#85) There are numerous instances of local food names that translate to multiple scientific food names.
Problem
There are 47 records documenting instances where a local food name (fl_local_food_name) is associated with more than one food scientific food name (fl_sci_food_name).
Bad Data
WITH duplicates AS (
SELECT
fl_sci_food_name,
COUNT(*) AS count
FROM clean.food_lookup
GROUP BY fl_sci_food_name
HAVING COUNT(*) > 1
)
SELECT
fl_local_food_name,
duplicates.fl_sci_food_name
FROM clean.food_lookup
JOIN duplicates ON clean.food_lookup.fl_sci_food_name = duplicates.fl_sci_food_name
ORDER BY fl_sci_food_name;
Notes
But see problem #87. This problem is only relevant if codes.food_names.description is derived from fl_sci_food_name (i.e., not fl_sci_food_name_gen); else, refer to #87.
SOLUTION
ALLOW FOR NOW, BUT NEEDS TO BE DISCUSSED AS A PI GROUP
9/3/2026 - ICG: Actually this returns cases where a scientific food name is associated with more than one local food name. That is, There can be multiple words for the same latin name.
Karl's comment
To allow, do some hackery when constructing the support tables. Make non-unique values unique by adding extra "duplicate #1 --" sort of text into the value of the column. Either "by hand" in the conversion code or by way of some algorythm.
2026-09-13 conversion decision
FOOD_NAMES.Description will be derived from
fl_sci_food_name_gen, not fl_sci_food_name. Problem
- 87 therefore controls description collisions for this conversion.
The historical query above does not test the relationship stated in the heading: it finds a scientific name shared by multiple local names. Do not use the historical count of 47 as a conversion assertion. If this issue is revisited, first replace the query with one that tests the intended relationship against the refreshed source.
(#86) There are numerous instances in the food_part_lookup that that conflate name, initial, and/or the english translation.
Problem
There are numerous instances in the food_part_lookup that that conflate name, initial, and/or the english translation. These issues are detailed below. Please note that this problem concerns the food_part_lookup table specifically.
Related, we need to clarify how we are treating the initials of food parts. Currently, initials are added to the codes.food_parts table with the corresponding food part name but with a trailing `-initial` so as not to violate the codes.food_parts.description unique constraint. So, both the food part name and food part initials are included in food events. This is probably not a good approach. Better would be to map the initial to the corresponding food part name and use only the full name (but then you do lose some reference to the original data). Ian needs to please consider.
Bad Data
query
WITH source_rows AS (
SELECT
ROW_NUMBER() OVER () AS src_ord,
COALESCE((
SELECT ARRAY_AGG(tok)
FROM (
SELECT BTRIM(x) AS tok
FROM REGEXP_SPLIT_TO_TABLE(COALESCE(fpl_local_food_part, ''), E'[;:,]') AS x
WHERE BTRIM(x) <> ''
) s
), ARRAY[]::text[]) AS local_arr,
COALESCE((
SELECT ARRAY_AGG(tok)
FROM (
SELECT BTRIM(x) AS tok
FROM REGEXP_SPLIT_TO_TABLE(COALESCE(fpl_food_part_initials, ''), E'[;:,]') AS x
WHERE BTRIM(x) <> ''
) s
), ARRAY[]::text[]) AS initials_arr,
COALESCE((
SELECT ARRAY_AGG(tok)
FROM (
SELECT BTRIM(x) AS tok
FROM REGEXP_SPLIT_TO_TABLE(COALESCE(fpl_english_food_part, ''), E'[;:,]') AS x
WHERE BTRIM(x) <> ''
) s
), ARRAY[]::text[]) AS english_arr
FROM clean.food_part_lookup
),
expanded AS (
SELECT
s.src_ord,
gs.idx,
COALESCE(s.local_arr[gs.idx], '') AS fpl_local_food_part,
COALESCE(s.initials_arr[gs.idx], '') AS fpl_food_part_initials,
COALESCE(s.english_arr[gs.idx], '') AS fpl_english_food_part
FROM source_rows s
CROSS JOIN LATERAL GENERATE_SERIES(
1,
GREATEST(
CARDINALITY(s.local_arr),
CARDINALITY(s.initials_arr),
CARDINALITY(s.english_arr)
)
) AS gs(idx)
),
deduped AS (
SELECT
e.*,
ROW_NUMBER() OVER (
PARTITION BY
e.fpl_local_food_part,
e.fpl_food_part_initials,
e.fpl_english_food_part
ORDER BY e.src_ord, e.idx
) AS rn
FROM expanded e
)
SELECT
fpl_local_food_part,
fpl_food_part_initials,
fpl_english_food_part
FROM deduped
WHERE rn = 1
ORDER BY src_ord, idx
fpl_local_food_part,
fpl_food_part_initials,
fpl_english_food_part;
summary
| fpl_local_food_part | fpl_food_part_initials | fpl_english_food_part |
|---|---|---|
| CHIPUKIZA | C | SHOOTS |
| MAJANI | J | LEAVES |
| MBEGU | MB | SEEDS |
| WADUDU | W | INSECTS |
| MABUA | B | PITH |
| MAGOMA | G | BARK |
| MATUNDA | T | FRUIT |
| MAUA | M | FLOWERS |
| UTOMVI | U | SAP |
| WADUDU WENGINE | D | INSECTS |
| UTOMVU | U | SAP |
| MCHWA | W | TERMITES |
| MIFUPA | NA | BONES |
| MITI | NA | TREE |
| MIZIZI | NA | ROOTS |
| NA | NA | NOT APPLICABLE |
| NONE | NA | None |
| NYAMA | N | MEAT |
| SIAFU | S | INSECTS |
| UNRECORDED | NA | UNRECORDED |
| WADUDU | D | INSECTS |
notes
- there are two initials (`W`, `D`) for `WADUDU` ~ `INSECTS`
- need to clarify `WADUDU WENGINE`, which also shares an initial (`D`) and english translation (`INSECTS`) as `WADUDU`
- the initial `W` is associated with both `WADUDU` and `MCHWA`
- different spellings for `SAP`: `UTOMVU` and `UTOMVI`
- `INSECTS` associated with `WADUDU`, `WADUDU WENGINE`, and `SIAFU`
- how should we treat `NA`, `NONE`, and `UNRECORDED`
SOLUTION
9/4/2026 Ian fixed mifupa, miti and mizizi in FOOD_BOUT by changing to UNRECORDED. DELETED FROM FOOD_PART_LOOKUP Consolidated insects to 'dudu' changed all "NA" to "None" Kept unrecorded fixed spellings of utomvi and chipukizi
I made all associated changes in FOOD_BOUT, choosing to use names rather than initials
(#87) There are numerous duplicate fl_sci_food_name_gen values in food_lookup.
Problem
There are 42 instances where fl_sci_food_name_gen values are associated with more than one fl_local_food_name value. This problem assumes that we are using the fl_sci_food_name_gen value for food_lookup.description (i.e., instead of fl_sci_food_name). If instead, food_lookup.description should reflect fl_sci_food_name then this particular problem is moot and can be ignored (but other problems will certainly arise with the switch to fl_sci_food_name).
Bad Data
WITH ranked AS (
SELECT
fl_local_food_name,
fl_sci_food_name,
fl_sci_food_name_gen,
COUNT(*) OVER (
PARTITION BY BTRIM(fl_sci_food_name_gen)
) AS sci_gen_count
FROM clean.food_lookup
WHERE fl_sci_food_name_gen IS NOT NULL
AND BTRIM(fl_sci_food_name_gen) <> ''
)
SELECT
fl_local_food_name,
fl_sci_food_name,
fl_sci_food_name_gen,
sci_gen_count
FROM ranked
WHERE sci_gen_count > 1
ORDER BY fl_sci_food_name_gen, fl_local_food_name, fl_sci_food_name;
Notes
But see problem #85. This problem is only relevant if codes.food_names.description is derived from fl_sci_food_name_gen (i.e., not fl_sci_food_name); else, refer to #85.
SOLUTION
ALLOW FOR NOW, DISCUSS AS A PI GROUP
2026-09-13 conversion decision
Use fl_sci_food_name_gen for
FOOD_NAMES.Description. Exclude lookup rows where
FL_exclude is true before deriving descriptions. Leave each
noncolliding generalized description unchanged. When one generalized
description is shared by multiple eligible local names, derive each target
description as:
fl_sci_food_name_gen || ' -- ' || fl_local_food_name
This representation is stable and preserves both source values. Do not use
mung's row-order-dependent numeric suffixes. In the 2026-09-13 snapshot, 39
generalized descriptions remained shared by 111 eligible local names after
applying FL_exclude. Rerun a corrected collision query as this
problem is encountered; the historical count of 42 predates the approved
exclusion rule.
* (#88) There are food_bout rows for which there is not a corresponding food name.
Problem
There are 2,114 clean.food_bout rows for which there is not a matching codes.food_names.name.
Bad data
Full record
Exact-match diagnostic: food_bout food names missing from codes.food_names
WITH food_rows AS (
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
1 AS seq,
fb_fl_local_food_name AS foodname
FROM clean.food_bout
-- WHERE fb_fl_local_food_name IS NOT NULL
UNION ALL
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
2 AS seq,
fb_local_food_name2 AS foodname
FROM clean.food_bout
-- WHERE NULLIF(fb_local_food_name2, '') IS NOT NULL
)
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
seq,
foodname
FROM food_rows
WHERE foodname NOT IN (
SELECT name
FROM codes.food_names
)
ORDER BY
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
seq,
foodname;
Summary of non-compliant values
Use this to list only the distinct food names present in clean.food_bout but missing from codes.food_names.
WITH food_rows AS (
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
1 AS seq,
fb_fl_local_food_name AS raw_foodname,
BTRIM(fb_fl_local_food_name) AS trimmed_foodname
FROM clean.food_bout
-- WHERE fb_fl_local_food_name IS NOT NULL
UNION ALL
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
2 AS seq,
fb_local_food_name2 AS raw_foodname,
BTRIM(fb_local_food_name2) AS trimmed_foodname
FROM clean.food_bout
-- WHERE NULLIF(fb_local_food_name2, '') IS NOT NULL
)
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
seq,
raw_foodname,
trimmed_foodname
FROM food_rows
WHERE raw_foodname NOT IN (
SELECT name
FROM codes.food_names
)
AND trimmed_foodname IN (
SELECT name
FROM codes.food_names
)
ORDER BY
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
seq,
raw_foodname;
white-space specific mismatches
Whitespace-only diagnostic: rows excluded only because of whitespace differences This finds values whose trimmed form exists in codes.food_names, but the raw value does not.
WITH food_rows AS (
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
1 AS seq,
fb_fl_local_food_name AS raw_foodname,
BTRIM(fb_fl_local_food_name) AS trimmed_foodname
FROM clean.food_bout
-- WHERE fb_fl_local_food_name IS NOT NULL
UNION ALL
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
2 AS seq,
fb_local_food_name2 AS raw_foodname,
BTRIM(fb_local_food_name2) AS trimmed_foodname
FROM clean.food_bout
-- WHERE NULLIF(fb_local_food_name2, '') IS NOT NULL
)
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
seq,
raw_foodname,
trimmed_foodname
FROM food_rows
WHERE raw_foodname NOT IN (
SELECT name
FROM codes.food_names
)
AND trimmed_foodname IN (
SELECT name
FROM codes.food_names
)
ORDER BY
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
seq,
raw_foodname;
SOLUTION
ALLOW FOR NOW. NEED TO CHECK WHETHER LEGIT NEW FOODS OR JUST POOR HANDWRITING, SPELLING, ETC
Karl's comments
Ok. Allowing means adding a bunch of odd values, that could continue to be used in the future until the data is cleaned and the "extra" codes removed.
(#89) There is a food_bout row that has both a compound fb_fpl_local_food_part value and where fb_fpl_local_food_part2 is not null.
Problem
| FB_FPL_local_food_part | FB_FPL_local_food_part2 |
|---|---|
| MATUNDA; CHIPUKIZA | MATUNDA; CHIPUKIZA |
SOLUTION
SHOULD BE ALLOWED
2026-09-13 conversion decision
Do not silently choose between a compound primary part and a conflicting explicit second part. If such a conflict remains after refreshed source cleanup, exclude the exact source row and document all source columns in the loader predicate.
The dated 2026-09-13 source contained one conflict:
| Date | Focal | Begin | End | Primary part | Primary name | Explicit second part | Second name |
|---|---|---|---|---|---|---|---|
| 1985-10-26 | EV | 11:22 | 11:51 | MATUNDA; CHIPUKIZI | MBULA | MATUNDA; CHIPUKIZI | BISHURUSHURU |
Under the approved parser, the compound second token is
CHIPUKIZI, while the explicit second value is the entire compound
MATUNDA; CHIPUKIZI. The older table above uses the stale spelling
CHIPUKIZA. Reproduce the complete current row exactly before
adding the exclusion; do not rely on this dated spelling or count.
* (#90) There are food_bout rows for which the fb_fpl_local_food_part or fb_fpl_local_food_part2 does not have a matching value in codes.food_parts.
Problem
There are 229 records for which a food_bout row does not have a match between for which the fb_fpl_local_food_part or fb_fpl_local_food_part2 and codes.food_parts. Note, however, that the record count is somewhat ambiguous since we are deailing with multiple parts (1 and 2), and multiple rows per event.
Bad data
Full record
diagnostic: fb_fpl_local_food_part or fb_fpl_local_food_part2 does not have a matching value in codes.food_parts
WITH prepared AS (
SELECT
fb.*,
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
) AS part1_norm,
UPPER(
NULLIF(
BTRIM(
split_part(
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
),
';',
1
)
),
''
)
) AS seq1_part,
CASE
WHEN regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
) LIKE '%;%'
THEN UPPER(
NULLIF(
BTRIM(
split_part(
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
),
';',
2
)
),
''
)
)
ELSE UPPER(NULLIF(BTRIM(fb.fb_fpl_local_food_part2), ''))
END AS seq2_part
FROM clean.food_bout fb
)
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
fb_fpl_local_food_part,
fb_fpl_local_food_part2,
seq1_part,
seq2_part,
CASE
WHEN seq1_part NOT IN (SELECT part FROM codes.food_parts)
THEN 'seq1_part missing from codes.food_parts'
WHEN seq2_part IS NOT NULL
AND seq2_part NOT IN (SELECT part FROM codes.food_parts)
THEN 'seq2_part missing from codes.food_parts'
END AS exclusion_reason
FROM prepared
WHERE seq1_part NOT IN (SELECT part FROM codes.food_parts)
OR (
seq2_part IS NOT NULL
AND seq2_part NOT IN (SELECT part FROM codes.food_parts)
)
ORDER BY
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time;
Summary of non-compliant values
WITH prepared AS (
SELECT
UPPER(
NULLIF(
BTRIM(
split_part(
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
),
';',
1
)
),
''
)
) AS seq1_part,
CASE
WHEN regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
) LIKE '%;%'
THEN UPPER(
NULLIF(
BTRIM(
split_part(
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
),
';',
2
)
),
''
)
)
ELSE UPPER(NULLIF(BTRIM(fb.fb_fpl_local_food_part2), ''))
END AS seq2_part
FROM clean.food_bout fb
)
SELECT missing_part, COUNT(*) AS occurrences
FROM (
SELECT seq1_part AS missing_part
FROM prepared
WHERE seq1_part NOT IN (SELECT part FROM codes.food_parts)
UNION ALL
SELECT seq2_part AS missing_part
FROM prepared
WHERE seq2_part IS NOT NULL
AND seq2_part NOT IN (SELECT part FROM codes.food_parts)
) x
GROUP BY missing_part
ORDER BY missing_part;
SOLUTION
ALLOW FOR NOW. NEED TO HAND CHECK AND DECIDE ON PROTOCOL
Karl's comments
Stevan can chime in here, but I think this means adding more illegitimate "legitimate" codes.
* (#91) There are food_bout rows that have fb_begin_feed_time > fb_end_feed_time.
Problem
There are 43 clean.food_bout rows that have fb_begin_feed_time > fb_end_feed_time.
Bad data
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
fb_begin_feed_time - fb_end_feed_time AS inverted_by,
fb_fpl_local_food_part,
fb_fl_local_food_name
FROM clean.food_bout
WHERE fb_begin_feed_time IS NOT NULL
AND fb_end_feed_time IS NOT NULL
AND fb_begin_feed_time > fb_end_feed_time
ORDER BY fb_fol_date, fb_fol_b_focal_animid, fb_begin_feed_time;
SOLUTION
IAN/VOLUNTEER TO FIX
* (#92) There are food_bout rows that do not have a corresponding follow.
Problem
clean.food_bout references follows by (fb_fol_b_focal_animid, fb_fol_date), but not all rows (n=455) match sokwedb.follows exactly.
Bad data
Includes both:
- Normalization-only matches: match after TRIM+UPPER (raw focal text quality issue).
- True misses: no matching follow even after normalization (actual source data gap).
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
fb_begin_feed_time - fb_end_feed_time AS inverted_by,
fb_fpl_local_food_part,
fb_fl_local_food_name
FROM clean.food_bout
WHERE fb_begin_feed_time IS NOT NULL
AND fb_end_feed_time IS NOT NULL
AND fb_begin_feed_time > fb_end_feed_time
ORDER BY fb_fol_date, fb_fol_b_focal_animid, fb_begin_feed_time;
SOLUTION
NEED TO LOOK AT THE SOURCE OF THE PROBLEM B/C NO-ONE ENTERS FEEDING FROM B-REC NOTES.
Karl's comments
I think it makes sense to normalize the data in the clean schema, upper casing and removing spaces. And then deal with what's left.
(#93) Need to address how we treat behaviours for the second value of food parts AND how they are derived.
Problem
1. In load_food_events.sql, we are loading into table food_events, where each row requires both:
- foodpart (NOT NULL + FK)
- foodname (NOT NULL + FK)
...but for seq=2, these cases behave differently:
- part2 exists, name2 missing: insert NODATA to name2
- part2 missing, name2 exists: insert NODATA to part2
- both missing: no seq=2 row.
- both exist: normal seq=2 row.
This approach required adding NODATA to codes.food_names and codes.food_parts.
Require approval for this approach or an altnernative.
2. Multiple food parts can be dervied in one of two ways:
- there are values for both fb_fpl_local_food_part and fb_fpl_local_food_part2, or
- there are multiple food parts in fb_fpl_local_food_part separated by a `;` (usually) or, rarely, `:`
Do we need to document or log if fb_fpl_local_food_part2 is derived from a compound fb_fpl_local_food_part value?
Bad data
count of values that meet these conditions
WITH prepared AS (
SELECT
fb.*,
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
) AS part1_norm,
UPPER(
NULLIF(
BTRIM(
split_part(
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
),
';',
2
)
),
''
)
) AS seq2_from_part1,
UPPER(NULLIF(BTRIM(fb.fb_fpl_local_food_part2), '')) AS seq2_from_part2,
NULLIF(BTRIM(fb.fb_local_food_name2), '') AS seq2_name
FROM clean.food_bout fb
),
effective AS (
SELECT
*,
CASE
WHEN part1_norm LIKE '%;%' THEN seq2_from_part1
ELSE seq2_from_part2
END AS seq2_part
FROM prepared
),
seq2_rows AS (
SELECT
COALESCE(seq2_part, 'NODATA') AS final_seq2_part,
COALESCE(seq2_name, 'NODATA') AS final_seq2_name
FROM effective
WHERE seq2_part IS NOT NULL
OR seq2_name IS NOT NULL
)
SELECT
CASE
WHEN final_seq2_part = 'NODATA' AND final_seq2_name = 'NODATA' THEN 'both_nodata'
WHEN final_seq2_part = 'NODATA' THEN 'part_nodata_only'
WHEN final_seq2_name = 'NODATA' THEN 'name_nodata_only'
ELSE 'no_nodata'
END AS seq2_case,
COUNT(*) AS rows
FROM seq2_rows
GROUP BY 1
ORDER BY 1;
full record
WITH prepared AS (
SELECT
fb.*,
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
) AS part1_norm,
UPPER(
NULLIF(
BTRIM(
split_part(
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
),
';',
2
)
),
''
)
) AS seq2_from_part1,
UPPER(NULLIF(BTRIM(fb.fb_fpl_local_food_part2), '')) AS seq2_from_part2,
NULLIF(BTRIM(fb.fb_local_food_name2), '') AS seq2_name
FROM clean.food_bout fb
),
effective AS (
SELECT
*,
CASE
WHEN part1_norm LIKE '%;%' THEN seq2_from_part1
ELSE seq2_from_part2
END AS seq2_part
FROM prepared
)
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_fpl_local_food_part,
fb_fpl_local_food_part2,
fb_fl_local_food_name,
fb_local_food_name2,
COALESCE(seq2_part, 'NODATA') AS final_seq2_part,
COALESCE(seq2_name, 'NODATA') AS final_seq2_name
FROM effective
WHERE seq2_part IS NOT NULL
OR seq2_name IS NOT NULL
ORDER BY
fb_fol_date,
fb_fol_b_focal_animid,
fb_fl_local_food_name;
SOLUTION
GET STEVAN TO EXPLAIN TO ICG AND ELO
2026-09-13 conversion decision
Add NODATA, described as No data recorded, to both
FOOD_NAMES and FOOD_PARTS. Create a Seq 2 row when
either approved secondary component exists. Use NODATA only for
the missing half of that pair:
- part exists, name missing: use
NODATAfor FoodName; - name exists, part missing: use
NODATAfor FoodPart; - neither exists: create no Seq 2 row; and
- both exist: preserve both approved values.
Normalize colon and semicolon delimiters and derive ordered compound tokens for
Seq 1 and Seq 2. If a compound-derived second part conflicts with an explicit
second part, exactly exclude the row under Problem #89. Do not silently apply
precedence. The dated source had 12 secondary names without an explicit second
part; these are eligible for the approved NODATA FoodPart rather
than exclusion. Rerun both queries above with the approved clean normalization
and exclusion projection as this problem is encountered.
* (#94) There are food_bout rows where the start or end of the observation is outside of allowable bounds.
Problem
There are 11 record where the food_bout is is outside of allowable bounds.
Bad data
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
CASE
WHEN fb_begin_feed_time IS NULL THEN 'BEGIN_NULL'
WHEN fb_begin_feed_time < '04:00:00'::time THEN 'BEGIN_BEFORE_MIN'
WHEN fb_begin_feed_time > '20:00:00'::time THEN 'BEGIN_AFTER_MAX'
ELSE 'BEGIN_OK'
END AS begin_status,
CASE
WHEN fb_end_feed_time IS NULL THEN 'END_NULL'
WHEN fb_end_feed_time < '04:00:00'::time THEN 'END_BEFORE_MIN'
WHEN fb_end_feed_time > '20:00:00'::time THEN 'END_AFTER_MAX'
ELSE 'END_OK'
END AS end_status,
COALESCE(
LEAST(GREATEST(fb_begin_feed_time, '04:00:00'::time), '20:00:00'::time),
'04:00:00'::time
) AS clamped_begin,
COALESCE(
LEAST(GREATEST(fb_end_feed_time, '04:00:00'::time), '20:00:00'::time),
'04:00:00'::time
) AS clamped_end
FROM clean.food_bout
WHERE fb_begin_feed_time IS NULL
OR fb_end_feed_time IS NULL
OR fb_begin_feed_time < '04:00:00'::time
OR fb_begin_feed_time > '20:00:00'::time
OR fb_end_feed_time < '04:00:00'::time
OR fb_end_feed_time > '20:00:00'::time
ORDER BY fb_fol_date, fb_fol_b_focal_animid, fb_begin_feed_time;
SOLUTION
ALLOW FOR NOW. IAN TO FIX
Karl's comment
See remarks on problem #81.
(#95) There is at least one food_bout rows where the start or stop time has a seconds value.
Problem
There is one food_bout row where the start or stop time has a seconds value, which violates our time-check constraints.
Bad data
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time
FROM clean.food_bout
WHERE EXTRACT(SECOND FROM fb_begin_feed_time) <> 0
OR EXTRACT(SECOND FROM fb_end_feed_time) <> 0
ORDER BY fb_fol_date, fb_fol_b_focal_animid, fb_begin_feed_time;
SOLUTION
IAN (HOPEFULLY) FIXED IN ACCESS 7/3
Karl's comment
Stevan, this is supposed to be fixed in the transition from the raw schema to the tidy schema. At least this is where the date and time intervals are normalized.
Maybe it got missed, or just needed bringing up as an issue. No real point in fixing it in MS Access. Doing so feels like losing information, and possibly being inconsistent in the way time intervals are normalized.
Update 2026-07-22 SRE cannot reproduce this error, e.g., see below querying the tidy schema, which returns zero records; calling this resolved.
SELECT
"FB_FOL_date",
"FB_FOL_B_focal_AnimID",
"FB_begin_feed_time",
"FB_end_feed_time"
FROM tidy."FOOD_BOUT"
WHERE EXTRACT(SECOND FROM "FB_begin_feed_time") <> 0
OR EXTRACT(SECOND FROM "FB_end_feed_time") <> 0
ORDER BY "FB_FOL_date", "FB_FOL_B_focal_AnimID", "FB_begin_feed_time";
(#96) There are GROOMINGS rows that have direction values that do not match to mappable values.
Problem
There are 26 GROOMINGS rows that have have a direction other than 'G', 'R', 'M', 'U', which are the only values that map to an allowable direction.
Bad data
Summary of non-compliant values
SELECT
COALESCE(grm_direction, '<NULL>') AS raw_direction,
quote_literal(grm_direction) AS quoted_direction,
UPPER(COALESCE(grm_direction, '')) AS upper_no_trim,
LENGTH(COALESCE(grm_direction, '')) AS raw_length,
COUNT(*) AS row_count
FROM clean.groom_bout
WHERE UPPER(COALESCE(grm_direction, '')) NOT IN ('G', 'R', 'M', 'U')
GROUP BY
COALESCE(grm_direction, '<NULL>'),
quote_literal(grm_direction),
UPPER(COALESCE(grm_direction, '')),
LENGTH(COALESCE(grm_direction, ''))
ORDER BY row_count DESC, raw_direction;
Full record
SELECT
grm_fol_date,
grm_fol_b_focal_animid,
grm_b_partner_animid,
grm_direction,
quote_literal(grm_direction) AS quoted_direction,
UPPER(COALESCE(grm_direction, '')) AS upper_no_trim,
LENGTH(COALESCE(grm_direction, '')) AS raw_length,
(
COALESCE(grm_direction, '') ~ '^[[:space:]]'
OR COALESCE(grm_direction, '') ~ '[[:space:]]$'
) AS has_edge_space,
grm_problems,
grm_comments,
grm_extracted_by
FROM clean.groom_bout
WHERE UPPER(COALESCE(grm_direction, '')) NOT IN ('G', 'R', 'M', 'U')
ORDER BY grm_fol_date, grm_fol_b_focal_animid, grm_b_partner_animid;
SOLUTION
ICG FIXED IN ACCESS 8/18/2026
(#97) There are GROOMINGS rows that have certainty values that do not match to mappable values.
Problem
There are 303 GROOMINGS rows that have have a direction other than 'Y' or 'N', which are the only values that map to an allowable certainty (Y -> 1; N -> 0). Only values of 0 and 1 are allowed in events.certainty.
Bad data
Summary of non-compliant values
SELECT
COALESCE(grm_time_certainty, '<NULL>') AS raw_certainty,
quote_literal(grm_time_certainty) AS quoted_certainty,
UPPER(COALESCE(grm_time_certainty, '')) AS upper_no_trim,
LENGTH(COALESCE(grm_time_certainty, '')) AS raw_length,
COUNT(*) AS row_count
FROM clean.groom_bout
WHERE UPPER(COALESCE(grm_time_certainty, '')) NOT IN ('Y', 'N')
GROUP BY
COALESCE(grm_time_certainty, '<NULL>'),
quote_literal(grm_time_certainty),
UPPER(COALESCE(grm_time_certainty, '')),
LENGTH(COALESCE(grm_time_certainty, ''))
ORDER BY row_count DESC, raw_certainty;
Full record
SELECT
grm_fol_date,
grm_fol_b_focal_animid,
grm_b_partner_animid,
grm_time_certainty,
quote_literal(grm_time_certainty) AS quoted_certainty,
UPPER(COALESCE(grm_time_certainty, '')) AS upper_no_trim,
LENGTH(COALESCE(grm_time_certainty, '')) AS raw_length,
(
COALESCE(grm_time_certainty, '') ~ '^[[:space:]]'
OR COALESCE(grm_time_certainty, '') ~ '[[:space:]]$'
) AS has_edge_space,
grm_problems,
grm_comments,
grm_extracted_by
FROM clean.groom_bout
WHERE UPPER(COALESCE(grm_time_certainty, '')) NOT IN ('Y', 'N')
ORDER BY grm_fol_date, grm_fol_b_focal_animid, grm_b_partner_animid;
SOLUTION
ICG FIXED IN ACCESS 8/18/2026
(#98) --- empty placeholder to restore count
(#99) There are GROOMINGS rows for which the extractedBy values is empty.
Problem
There are 10,710 GROOMINGS rows for which the extractedBy values is empty. This value cannot be NULL.
Bad data
SELECT
grm_fol_date,
grm_fol_b_focal_animid,
grm_b_partner_animid,
grm_direction,
grm_time_certainty,
grm_extracted_by,
quote_literal(grm_extracted_by) AS quoted_extracted_by,
LENGTH(COALESCE(grm_extracted_by, '')) AS raw_length,
grm_problems,
grm_comments
FROM clean.groom_bout
WHERE NULLIF(BTRIM(grm_extracted_by), '') IS NULL
ORDER BY grm_fol_date, grm_fol_b_focal_animid, grm_b_partner_animid;
SOLUTION
ICG CHANGED ALL NULLS TO 'UNK' IN ACCESS. 8/18/2026
* (#100) There are GROOMINGS rows that have extractedby values that do not match to mappable people values.
Problem
There are 14,771 GROOMINGS rows that have extractedby values that do not match to mappable codes.people values.
Bad data
Summary of non-compliant values
SELECT
BTRIM(gb.grm_extracted_by) AS normalized_extracted_by,
COUNT(*) AS row_count
FROM clean.groom_bout gb
WHERE NULLIF(BTRIM(gb.grm_extracted_by), '') IS NOT NULL
AND NOT EXISTS (
SELECT 1
FROM codes.people p
WHERE p.person = BTRIM(gb.grm_extracted_by)
)
GROUP BY BTRIM(gb.grm_extracted_by)
ORDER BY row_count DESC, normalized_extracted_by;
Full record
SELECT
gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_b_partner_animid,
gb.grm_extracted_by,
BTRIM(gb.grm_extracted_by) AS normalized_extracted_by,
gb.grm_direction,
gb.grm_time_certainty,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout gb
WHERE NULLIF(BTRIM(gb.grm_extracted_by), '') IS NOT NULL
AND NOT EXISTS (
SELECT 1
FROM codes.people p
WHERE p.person = BTRIM(gb.grm_extracted_by)
)
ORDER BY gb.grm_fol_date, gb.grm_fol_b_focal_animid, gb.grm_b_partner_animid;
* (#101) There are AGGRESSION_EVENT rows that have NULL focals, a NULL ae_fol_b_focal_id value
Problem
There are 144 aggression_event rows that have NULL for the focal.
Bad data
SELECT * FROM easy.aggression_event WHERE ae_fol_b_focal_id IS NULL;
Solution
Change these values to the unknown individual, UNK.
This means that these rows are converted to WATCHES rows with a Type of B, and UNK as the focal.
The data needs to be reviewed to see if this is appropriate. See also problem #102.
* (#102) There are AGGRESSION_EVENT rows that have NULL times, a NULL ae_time value
Problem
There are 107 aggression_event rows that have NULL for the time. Note that all of these rows also have NULL for the focal, problem #101.
Bad data
SELECT * FROM easy.aggression_event WHERE ae_time IS NULL;
Solution
Change these values to the unknown time, midnight, 00:00.
These need to be reviewed to ensure the change is appropriate.
Meeting discussions indicate the many, ideally all, of these rows are the result of perusal of a book's content, where aggressions were mentioned but these aggressions did not otherwise appear in the data.
(#103) There are BIOGRAPHY rows that have empty birthgroup values.
Problem
There are 64 BIOGRAPHY rows that have empty strings for birthgroup; NULL values are allowed but not empty strings. Some of the offending records are addressed also in Problem #13.
Bad data
note: (temporary) solution applied to clean so query tidy
SELECT "B_AnimID",
"B_AnimName",
quote_nullable("B_BirthGroup") AS birthgroup,
length("B_BirthGroup") AS character_length,
octet_length("B_BirthGroup") AS byte_length,
"B_BGCertainty"
FROM tidy."BIOGRAPHY"
WHERE "B_BirthGroup" IS NOT NULL
AND BTRIM("B_BirthGroup") = ''
ORDER BY "B_AnimID";
Solution
Propose converting empty strings to NULL in the clean schema, PI to confirm.
UPDATE biography SET b_birthgroup = NULLIF(BTRIM(b_birthgroup), '') WHERE b_birthgroup IS DISTINCT FROM NULLIF(BTRIM(b_birthgroup), '');
Addressed with commit f2ddb85cd22d3e5233d049d92ae8cb361a538114
(#104) There are BIOGRAPHY rows that have empty momid values.
Problem
There are 89 BIOGRAPHY rows that have empty strings for momid; NULL values are allowed but not empty strings.
Bad data
note: (temporary) solution applied to clean so query tidy
SELECT "B_AnimID",
"B_AnimName",
quote_nullable("B_MomID") AS momid,
length("B_MomID") AS character_length,
octet_length("B_MomID") AS byte_length,
"B_Birthdate"
FROM tidy."BIOGRAPHY"
WHERE "B_MomID" IS NOT NULL
AND BTRIM("B_MomID") = ''
ORDER BY "B_AnimID";
Solution
Propose converting empty strings to NULL in the clean schema, PI to confirm.
UPDATE biography SET b_momid = NULLIF(BTRIM(b_momid), '') WHERE b_momid IS DISTINCT FROM NULLIF(BTRIM(b_momid), '');
Addressed with commit 5e0e787825e847eac1cc93375e26ea1d3f715d3e
(#105) There is a comm member log record that is missing a description.
Problem
There is a single comm member log records where the description is missing.
Bad data
note: querying tidy
SELECT date_of_update,
chimp_id,
quote_nullable(update_description) AS update_description,
length(update_description) AS description_length,
update_rationale,
"made by"
FROM tidy."COMMUNITY_MEMBERSHIP_UPDATE_LOG"
WHERE update_description IS NULL
OR BTRIM(update_description) = ''
ORDER BY date_of_update,
chimp_id,
update_rationale,
"made by";
solution
Fixed in Access by ICG - duplicated update_rationale
(#106) There are GROOM_SCANS records whose extracted-by person is not in PEOPLE.
Problem
There are 44,679 GROOM_SCANS records where the GS_extracted_by value is not a PEOPLE.Person value. All GROOM_SCANS records are affected. There are four distinct offending values: KASEN, KAREN MCLELLAN, Anika Richter, and Janelle Carmichael.
Bad data
note: querying clean, the schema read by the production loader
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
quote_nullable(gs.gs_extracted_by) AS gs_extracted_by,
length(gs.gs_extracted_by) AS extracted_by_length
FROM clean.groom_scans AS gs
WHERE NOT EXISTS (
SELECT 1
FROM codes.people AS people
WHERE people.person = gs.gs_extracted_by)
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
Solution
During construction of the clean schema, add one deterministic spelling of each case-insensitive GROOM_SCANS extractor value to clean.people when no case-insensitive PEOPLE match already exists. Prefer an existing PEOPLE spelling, and otherwise prefer a mixed-case source spelling over an all-uppercase or all-lowercase spelling. Rewrite gs_extracted_by to the exact spelling stored in clean.people.
The ordinary support-table loader copies these rows to codes.people with active set to true before B-record groom scans are loaded. This preserves all 44,679 source rows and leaves the production foreign key and active-person rule intact.
* (#107) There are GROOM_SCAN_AREC records having the same chimpanzee as both grooming participants.
Problem
There are 1,784 GROOM_SCAN_AREC records where Chimp_1 and Chimp_2 are the same individual. The records span 210 dates and 39 distinct chimpanzees, and all have a Direction value of M. The production loader converts Direction M into two Mutual ROLES rows. These records cannot be loaded because a participant may occur only once in an event and the attendance groom-scan rules require two different participants. The project investigators must determine whether these records represent self-grooming or erroneous participant data and how they should be represented in the production database.
Bad data
note: querying clean, the schema read by the production loader
SELECT gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
gs.direction
FROM clean.groom_scan_arec AS gs
WHERE gs.chimp_1 = gs.chimp_2
ORDER BY gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2;
* (#108) There are GROOM_SCAN_AREC participants that are not in BIOGRAPHY_DATA.
Problem
There are 1,124 GROOM_SCAN_AREC records having at least one participant that is not a BIOGRAPHY_DATA.AnimID value. There are 81 distinct missing participant values. The missing value occurs in Chimp_1 in 112 records and in Chimp_2 in 1,016 records; four records have missing values in both columns. None of the 81 values are present in clean.BIOGRAPHY, from which BIOGRAPHY_DATA is loaded. The production loader cannot create the related ROLES rows until the project investigators determine whether these values identify individuals that should be added to BIOGRAPHY, corrected to existing AnimID values, or otherwise represented.
Bad data
note: querying clean, the schema read by the production loader, and sokwedb.BIOGRAPHY_DATA, the referenced production table
SELECT gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
gs.direction,
bad_participant.source_column,
quote_nullable(bad_participant.animid) AS missing_animid,
length(bad_participant.animid) AS animid_length
FROM clean.groom_scan_arec AS gs
CROSS JOIN LATERAL (
VALUES ('Chimp_1', gs.chimp_1),
('Chimp_2', gs.chimp_2)
) AS bad_participant(source_column, animid)
WHERE NOT EXISTS (
SELECT 1
FROM sokwedb.biography_data AS bio
WHERE bio.animid = bad_participant.animid)
ORDER BY gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
bad_participant.source_column;
SOLUTION
HAVE A STUDENT CHECK AGAINST PAPER
* (#109) There are GROOM_SCAN_AREC records whose dates have no ATTENDANCE records from which to obtain a community.
Problem
There are 5,111 GROOM_SCAN_AREC records on 134 dates for which there is no ATTENDANCE record on the same date. GROOM_SCAN_AREC does not contain a community value, and the production loader obtains the WATCHES.CommID value from ATTENDANCE.A_CL_Community_ID by date. Consequently, the loader cannot create the required WATCHES row for these scans without assuming a community. After the temporary exclusions for Problems #107 and #108, 4,786 otherwise-loadable records on 133 dates are affected. The project investigators must determine the appropriate community for these scan dates.
Bad data
note: querying clean, the schema read by the production loader
SELECT gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
gs.direction
FROM clean.groom_scan_arec AS gs
WHERE NOT EXISTS (
SELECT 1
FROM clean.attendance AS attendance
WHERE attendance.a_date = gs.date)
ORDER BY gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2;
* (#110) There are GROOM_SCAN_AREC records involving individuals after their departure from the study.
Problem
There are 2,090 GROOM_SCAN_AREC records where at least one participant's scan date is after the individual's BIOGRAPHY_DATA.DepartDate. The records involve 18 distinct individuals on 177 dates. The offending individual occurs in Chimp_1 in 1,928 records and in Chimp_2 in 162 records; no record has both participants after their departure dates. After the temporary exclusions for Problems #107, #108, and #109, 1,725 otherwise-loadable records on 147 dates are affected. The production schema requires event participants to be under study on the WATCHES.Date, so these records cannot be loaded until the project investigators determine whether the scan participants or biography departure dates should be corrected.
The trigger error originally reported PT's EntryDate, 1970-09-11, as its DepartDate due to an error in the trigger's diagnostic query. PT's actual BIOGRAPHY_DATA.DepartDate is 1973-04-17, which is before the first affected PT scan on 1973-05-25. The constraint comparison itself used the correct DepartDate and therefore rejected the row correctly.
Bad data
note: querying clean, the schema read by the production loader, and sokwedb.BIOGRAPHY_DATA, the referenced production table
SELECT gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
gs.direction,
bad_participant.source_column,
bad_participant.animid,
bio.departdate,
gs.date - bio.departdate AS days_after_departure
FROM clean.groom_scan_arec AS gs
CROSS JOIN LATERAL (
VALUES ('Chimp_1', gs.chimp_1),
('Chimp_2', gs.chimp_2)
) AS bad_participant(source_column, animid)
JOIN sokwedb.biography_data AS bio
ON bio.animid = bad_participant.animid
WHERE gs.date > bio.departdate
ORDER BY gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
bad_participant.source_column;
* (#111) There are GROOM_SCAN_AREC records involving individuals before their entry into the study.
Problem
There are 130 GROOM_SCAN_AREC records where at least one participant's scan date is before the individual's BIOGRAPHY_DATA.EntryDate. The records involve 10 distinct individuals on 21 dates. The offending individual occurs in Chimp_1 in 37 records and in Chimp_2 in 93 records; no record has both participants before their entry dates. After the temporary exclusions for Problems #107 through #110, 30 otherwise-loadable records on 11 dates are affected. The production schema requires event participants to be under study on the WATCHES.Date, so these records cannot be loaded until the project investigators determine whether the scan participants or biography entry dates should be corrected.
Bad data
note: querying clean, the schema read by the production loader, and sokwedb.BIOGRAPHY_DATA, the referenced production table
SELECT gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
gs.direction,
bad_participant.source_column,
bad_participant.animid,
bio.entrydate,
bio.entrydate - gs.date AS days_before_entry
FROM clean.groom_scan_arec AS gs
CROSS JOIN LATERAL (
VALUES ('Chimp_1', gs.chimp_1),
('Chimp_2', gs.chimp_2)
) AS bad_participant(source_column, animid)
JOIN sokwedb.biography_data AS bio
ON bio.animid = bad_participant.animid
WHERE gs.date < bio.entrydate
ORDER BY gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
bad_participant.source_column;
(#112) GROOM_BOUT uses U as an unknown initiator or terminator animal ID
Problem
Some GROOM_BOUT rows contain U in GRM_B_initiator_AnimId or GRM_B_terminator_AnimId. Although U is a valid value for GRM_direction, it is not a valid animal ID.
The production GROOMINGS.Initiator and GROOMINGS.Terminator columns reference participant rows in ROLES. During conversion, the loader creates roles for the focal and partner animals and then looks up the role corresponding to the recorded initiator or terminator. A value of U cannot match either participant, causing the loader to fail with query returned no rows.
This problem became visible after the Problem #99 and Problem #100 exclusions were removed. Those exclusions had prevented many affected grooming bouts from reaching the initiator and terminator lookup.
Bad data
The following query identifies every occurrence of `U` in the initiator and terminator animal-ID fields:
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
gb.grm_direction,
invalid_participant.source_column,
invalid_participant.animid AS offending_animid,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
CROSS JOIN LATERAL (
VALUES ('GRM_B_initiator_AnimId', gb.grm_b_initiator_animid),
('GRM_B_terminator_AnimId', gb.grm_b_terminator_animid)
) AS invalid_participant(source_column, animid)
WHERE BTRIM(invalid_participant.animid) = 'U'
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
invalid_participant.source_column;
Solution
During construction of the clean schema, convert U in groom_bout.grm_b_initiator_animid and groom_bout.grm_b_terminator_animid to SQL NULL. Do not change groom_bout.grm_direction when it contains U, because that is a valid direction code representing unknown grooming direction.
The grooming loader already treats a NULL initiator or terminator as unknown and stores NULL in the corresponding production column without attempting to find a participant role.
Other nonempty initiator or terminator animal IDs that do not identify either the focal or partner animal should remain conversion errors and be reviewed separately rather than being converted automatically to NULL.
Approved by the investigator on 2026-09-11.
(#113) The Access BIOGRAPHY.NONE row lacks values required by BIOGRAPHY_DATA
Problem
The A-record groom-scan conversion represents feeding-station scans as A-record WATCHES. These scans have no focal individual, so the conversion uses the special AnimID NONE for the related WATCHES row rather than incorrectly identifying the focal individual as unknown with UNK. The production BIOGRAPHY_DATA table must therefore contain a NONE row.
The conversion originally created this row in clean.BIOGRAPHY. After the row was added to the Access BIOGRAPHY table, retaining that INSERT caused the clean-schema build to fail with a duplicate BIOGRAPHY primary key for NONE. Removing the INSERT exposed a second problem: the Access row contains NULL in fields that are required by BIOGRAPHY_DATA, beginning with BCCertainty, so it cannot be loaded directly into the production table.
Bad data
The affected row is the single Access BIOGRAPHY record whose B_AnimID is NONE. A diagnostic query is unnecessary because this is an intentionally defined sentinel record rather than an unidentified set of source-data rows.
Solution
Keep Access authoritative for the existence of the NONE BIOGRAPHY row. During construction of the clean schema, update only that row with the previously established no-focal placeholder values required by BIOGRAPHY_DATA. Do not insert a second row, modify the restored Access data in raw, or apply the correction directly to sokwedb.
This preserves the decision that A-record groom scans use NONE when no focal individual exists while allowing the Access-supplied sentinel record to satisfy the production biography constraints.
(#114) BRECORD_NOTES times are not uniformly compatible with EVENTS
Problem
The Access BRECORD_NOTES.BREC_time column is stored as a timestamp and is converted to a PostgreSQL TIME value in the tidy schema. Problem #10 corrected records whose timestamp had the wrong date component, but converting the datatype preserves seconds. The production EVENTS.Start and EVENTS.Stop columns record times to the minute and reject nonzero seconds.
Some B-record notes also use midnight to indicate that no time was recorded. The production schema previously reserved the sdb_no_time midnight sentinel for aggression events, so it rejected otherwise-loadable B-record notes with no recorded time.
In the local conversion data examined on 2026-08-20, 22 rows contained nonzero seconds, 46 rows contained midnight, and no rows had a SQL NULL time. Forty-five of the midnight rows had a matching follow-derived B-record watch.
Bad data
SELECT brec_fol_date,
brec_fol_b_focal_animid,
brec_time,
extract(second FROM brec_time) AS seconds
FROM clean.brecord_notes
WHERE brec_time IS NULL
OR brec_time = '00:00'::TIME
OR extract(second FROM brec_time) <> 0
ORDER BY brec_fol_date,
brec_fol_b_focal_animid,
brec_time;
Solution
Truncate nonzero seconds from BREC_time in tidy_cleanups.sql before the column is converted to TIME. Do not round the values. Convert a SQL NULL time to sdb_no_time at the production load boundary and preserve an existing midnight value as that sentinel. Permit both aggression and B-record-note EVENTS rows to use sdb_no_time, with both Start and Stop set to the sentinel.
* (#115) BRECORD_NOTES rows lack a matching follow-derived WATCHES row
Problem
A production BRECORD_NOTES row belongs to an EVENTS row, which in turn must belong to a WATCHES row. The ordinary B-record watches are created from clean.follow. Some B-record notes have no follow with the same focal individual and date. It is not yet known which records truly lack a follow and which contain an incorrect focal or date.
Creating provisional watches would require guessing the focal, community, or date and would make the affected records difficult to identify for later review. In the local conversion data examined on 2026-08-20, 35,517 of 603,977 B-record-note rows lacked a matching follow.
Bad data
SELECT brecord_notes.*
FROM clean.brecord_notes
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow
WHERE follow.fol_b_animid =
brecord_notes.brec_fol_b_focal_animid
AND follow.fol_date = brecord_notes.brec_fol_date)
ORDER BY brec_fol_date,
brec_fol_b_focal_animid,
brec_time;
Solution
Temporarily exclude these rows from the production conversion. Retain them unchanged in clean.brecord_notes so that the focal and date can be reviewed and corrected in Access. The loader must reuse only an existing type-B WATCHES row having the same focal and date; it must not create a watch for an unmatched B-record note.
* (#116) BRECORD_NOTES contains times outside the EVENTS limits
Problem
Except for the sdb_no_time sentinel, production event times must be between sdb_min_event_start (04:00) and sdb_max_event_stop (20:00), inclusive. Some B-record-note times are outside those limits and their correct values must be determined from the source records.
In the local conversion data examined on 2026-08-20, 751 rows had an out-of-range time other than midnight. Of those, 711 had a matching follow-derived watch and would otherwise be loadable.
Bad data
SELECT brecord_notes.*
FROM clean.brecord_notes
WHERE brec_time IS NOT NULL
AND brec_time <> '00:00'::TIME
AND (brec_time < '04:00'::TIME
OR brec_time > '20:00'::TIME)
ORDER BY brec_fol_date,
brec_fol_b_focal_animid,
brec_time;
Solution
Temporarily exclude the offending rows from the production conversion while retaining them in clean.brecord_notes. Correct the source times in Access after reviewing the original records. Do not clamp the values or weaken the production time constraints.
(#117) BRECORD_NOTES text values do not satisfy production constraints
Problem
All production BRECORD_NOTES text columns are NOT NULL. Observation must also be trimmed of leading and trailing spaces. The remaining text columns permit the empty string to represent absent text but reject values consisting only of whitespace.
The Access source uses SQL NULL for absent text and contains some whitespace-only values. In the local conversion data examined on 2026-08-20, the rows otherwise eligible for conversion included 41 NULL observations and 14,349 untrimmed observations. The optional columns contained between 15,080 and 566,253 NULL values. Four comments, three Voc values, and three VocID values consisted only of whitespace.
Bad data
SELECT brec_fol_date,
brec_fol_b_focal_animid,
brec_time,
field_name,
quote_nullable(field_value) AS field_value
FROM clean.brecord_notes
CROSS JOIN LATERAL (
VALUES ('Observation', brec_observation),
('Comments', brec_comments),
('Observer', brec_observer),
('Translator', brec_translator),
('TranscribedBy', brec_transcribed_by),
('Voc', brec_voc),
('VocID', brec_vocid),
('GroomingAggression', brec_grooming_aggressionflag),
('Duplicate', brec_duplicate_flag)
) AS source_text(field_name, field_value)
WHERE field_value IS NULL
OR (field_value <> '' AND btrim(field_value) = '')
OR (field_name = 'Observation'
AND field_value <> btrim(field_value))
ORDER BY brec_fol_date,
brec_fol_b_focal_animid,
brec_time,
field_name;
Solution
Normalize only what the production constraints require at the final load boundary. Convert a NULL or whitespace-only value to the empty string. Trim Observation because its production column requires it, but preserve leading and trailing spaces in other nonempty source text. Keep the unchanged source-like values available in clean.brecord_notes.
(#118) PANTGRUNT_EVENT rows have no recorded time
Problem
Some PANTGRUNT_EVENT rows have NULL in pg_time. EVENTS.Start and EVENTS.Stop are not nullable, but the absence of a recorded pantgrunt time is meaningful and must not cause the entire row to be discarded. In the local conversion data examined on 2026-08-21, 684 rows had no recorded time.
Bad data
SELECT *
FROM clean.pantgrunt_event
WHERE pg_time IS NULL
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_b_actor_id,
pg_b_recipient_id;
Solution
At the production load boundary, convert a NULL pg_time to the sdb_no_time value, midnight, and use that value for both EVENTS.Start and EVENTS.Stop. Extend the EVENTS constraints so pantgrunt events, like aggression events, may use sdb_no_time. Retain NULL in clean.pantgrunt_event so the clean schema continues to distinguish missing source data from an actual recorded time.
(#119) PANTGRUNT_EVENT times may contain seconds
Problem
EVENTS times are stored to the minute and reject nonzero seconds. The Access pg_time value is initially represented as a timestamp, and merely converting it to a PostgreSQL time value does not remove seconds. The local clean data examined on 2026-08-21 contained no remaining nonzero seconds, but the pantgrunt workflow previously had no explicit normalization guaranteeing that result.
Bad data
SELECT pg_date,
pg_fol_b_foc_id,
pg_time,
extract(second FROM pg_time) AS seconds
FROM clean.pantgrunt_event
WHERE extract(second FROM pg_time) <> 0
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time;
Solution
During construction of the tidy schema, truncate pg_time to the minute before tidy.sql converts the column to TIME. Do not round the value. The pantgrunt sanity check rejects the conversion if a value containing seconds nevertheless reaches the clean schema.
* (#120) PANTGRUNT_EVENT times fall outside the permitted event window
Problem
Except for sdb_no_time, production event times must be between 04:00 and 20:00, inclusive. In the local conversion data examined on 2026-08-21, 35 non-NULL pantgrunt times were outside this interval. Their correct values cannot be inferred during conversion.
Bad data
SELECT *
FROM clean.pantgrunt_event
WHERE pg_time IS NOT NULL
AND (pg_time < '04:00'::TIME
OR pg_time > '20:00'::TIME)
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time,
pg_b_actor_id,
pg_b_recipient_id;
Solution
Temporarily exclude these rows from the production conversion and report their count in the pantgrunt sanity check. Retain the rows unchanged in clean.pantgrunt_event. The project investigators must review and correct the source times in Access; the conversion must not clamp them or weaken the production time constraints.
* (#121) PANTGRUNT_EVENT rows name the same actor and recipient
Problem
A production pantgrunt is a dyadic event involving two different participants. Some source rows identify the same animal as both pg_b_actor_id and pg_b_recipient_id. In the local conversion data examined on 2026-08-21, 11 rows had the same trimmed actor and recipient value.
Bad data
SELECT *
FROM clean.pantgrunt_event
WHERE BTRIM(pg_b_actor_id) = BTRIM(pg_b_recipient_id)
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time,
pg_b_actor_id;
Solution
Temporarily exclude these rows and report their count in the pantgrunt sanity check. The project investigators must determine the correct actor or recipient and correct the source records in Access.
(#122) PANTGRUNT_EVENT rows have no data source
Problem
PANTGRUNTS.Source is required and must reference PG_SOURCES, but some source pantgrunt rows have a NULL, empty, or whitespace-only pg_data_source. In the local conversion data examined on 2026-08-21, 109 rows had no data source.
Bad data
SELECT *
FROM clean.pantgrunt_event
WHERE NULLIF(BTRIM(pg_data_source), '') IS NULL
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time,
pg_b_actor_id,
pg_b_recipient_id;
Solution
During construction of the clean schema, normalize source codes by trimming and converting them to uppercase. Convert a missing source to NODATA. Populate PG_SOURCES from the distinct normalized source values and give NODATA the description "No pantgrunt data source was recorded". Load NODATA into PANTGRUNTS.Source for affected rows.
(#123) PANTGRUNT_EVENT rows have no extractor
Problem
PANTGRUNTS.EnteredBy is required and must reference an active PEOPLE row, but some source rows have a NULL, empty, or whitespace-only pg_extracted_by value. In the local conversion data examined on 2026-08-21, 2,223 rows had no recorded extractor.
Bad data
SELECT *
FROM clean.pantgrunt_event
WHERE NULLIF(BTRIM(pg_extracted_by), '') IS NULL
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time,
pg_b_actor_id,
pg_b_recipient_id;
Solution
During construction of the clean schema, change a NULL, empty, or whitespace-only pg_extracted_by value to UNK. The established UNK row is copied from clean.people to codes.people as an active person and is loaded into PANTGRUNTS.EnteredBy for affected eligible rows.
(#124) PANTGRUNT_EVENT extractor values are missing from PEOPLE
Problem
Many nonempty pg_extracted_by values do not exactly match a clean.people person value. The local conversion data examined on 2026-08-21 had 26,334 rows without an exact PEOPLE match when missing values were included, and 24,111 nonmissing rows without a match after trimming. Some extractor names also occur with case variants, such as Anne and ANNE, while codes.people requires case-insensitive uniqueness.
Bad data
SELECT BTRIM(pg.pg_extracted_by) AS extracted_by,
COUNT(*) AS row_count,
ARRAY_AGG(DISTINCT pg.pg_extracted_by ORDER BY pg.pg_extracted_by) AS source_spellings
FROM clean.pantgrunt_event AS pg
WHERE NULLIF(BTRIM(pg.pg_extracted_by), '') IS NOT NULL
AND NOT EXISTS (
SELECT 1
FROM clean.people AS p
WHERE p.person = BTRIM(pg.pg_extracted_by))
GROUP BY BTRIM(pg.pg_extracted_by)
ORDER BY row_count DESC,
extracted_by;
Solution
During construction of the clean schema, trim nonempty pantgrunt extractor values and add one deterministic spelling of each case-insensitive value to clean.people when no case-insensitive PEOPLE match already exists. Prefer an existing PEOPLE spelling, and otherwise prefer a mixed-case source spelling over an all-uppercase or all-lowercase spelling. Rewrite pg_extracted_by to the exact spelling stored in clean.people. The ordinary support-table loader then copies these rows to codes.people with active set to true, which is required because new PANTGRUNTS rows may reference only active people.
(#125) PANTGRUNT_EVENT two-sided flags use multiple encodings
Problem
The pg_two_sided_flag determines whether the two participants receive directed Actor and Actee roles or two Mutual roles. The Access data use NULL, N, Y, X, lowercase x, and potentially questionable yes/no forms rather than a single canonical encoding.
Bad data
SELECT pg_two_sided_flag,
quote_nullable(pg_two_sided_flag) AS quoted_value,
COUNT(*) AS row_count
FROM clean.pantgrunt_event
GROUP BY pg_two_sided_flag
ORDER BY row_count DESC,
pg_two_sided_flag;
Solution
Normalize NULL, the empty string, N, N?, and ? to N in the clean schema. Normalize Y, Y?, X, and lowercase x to Y. During loading, Y creates two Mutual roles; N creates an Actor role for pg_b_actor_id and an Actee role for pg_b_recipient_id. The sanity check stops conversion if an unrecognized value remains.
(#126) PANTGRUNT_EVENT multiple-participant flags use noncanonical encodings
Problem
PANTGRUNTS.MultiActors and PANTGRUNTS.MultiRecipients are required Boolean values, while the corresponding Access fields are nullable text flags. The local conversion data include a lowercase x in pg_multiple_recipient_flag, and future dumps may include questionable yes/no forms.
Bad data
SELECT flag_name,
flag_value,
COUNT(*) AS row_count
FROM clean.pantgrunt_event
CROSS JOIN LATERAL (
VALUES ('pg_multiple_actor_flag', pg_multiple_actor_flag),
('pg_multiple_recipient_flag', pg_multiple_recipient_flag)
) AS flags(flag_name, flag_value)
GROUP BY flag_name,
flag_value
ORDER BY flag_name,
row_count DESC,
flag_value;
Solution
Normalize NULL, the empty string, N, N?, and ? to the empty string in the clean schema and load them as false. Normalize Y, Y?, X, and lowercase x to X and load them as true. The sanity check stops conversion if an unrecognized value remains.
* (#127) PANTGRUNT_EVENT participants are missing from BIOGRAPHY_DATA
Problem
Every pantgrunt actor and recipient must be represented by a BIOGRAPHY_DATA row before a corresponding ROLES row can be created. In the local conversion data examined on 2026-08-21, 165 rows had an actor missing from BIOGRAPHY_DATA and 609 rows had a recipient missing from BIOGRAPHY_DATA. A row can be counted in both categories.
Bad data
SELECT pg.*,
participant.source_column,
participant.animid
FROM clean.pantgrunt_event AS pg
CROSS JOIN LATERAL (
VALUES ('pg_b_actor_id', pg.pg_b_actor_id),
('pg_b_recipient_id', pg.pg_b_recipient_id)
) AS participant(source_column, animid)
WHERE NOT EXISTS (
SELECT 1
FROM clean.biography AS b
WHERE b.b_animid = BTRIM(participant.animid))
ORDER BY pg.pg_date,
pg.pg_fol_b_foc_id,
pg.pg_time,
participant.source_column,
participant.animid;
Solution
Temporarily exclude any row whose trimmed actor or recipient does not have a matching BIOGRAPHY_DATA animal. Report actor and recipient counts separately in the sanity check. The project investigators must identify the intended individuals and correct the source records in Access.
(#128) PANTGRUNT_EVENT focal/date keys have no corresponding follow
Problem
A pantgrunt event must relate to a WATCHES row. The normal B-record watches are created from clean.follow, but many pantgrunt focal/date keys have no matching follow. The local conversion data examined on 2026-08-21 contained 2,443 normalized focal/date keys without a B-record watch before other pantgrunt exclusions were applied.
Bad data
SELECT COALESCE(NULLIF(BTRIM(pg.pg_fol_b_foc_id), ''), 'NONE') AS watch_animid,
pg.pg_date,
COUNT(*) AS pantgrunt_rows
FROM clean.pantgrunt_event AS pg
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow AS f
WHERE f.fol_b_animid = BTRIM(pg.pg_fol_b_foc_id)
AND f.fol_date = pg.pg_date)
GROUP BY COALESCE(NULLIF(BTRIM(pg.pg_fol_b_foc_id), ''), 'NONE'),
pg.pg_date
ORDER BY pg.pg_date,
watch_animid;
Solution
Prefer and reuse an existing B-record WATCHES row having the same focal and date. If no such watch exists for an otherwise eligible pantgrunt, reuse an existing Other watch or create a WATCHES row with Type Other. Use the pantgrunt community for a newly created watch. This avoids falsely representing an ad-hoc observation as a follow and avoids the missing-arrival warnings associated with B-record watches. The source association can be corrected later without discarding the converted pantgrunt.
(#129) PANTGRUNT_EVENT rows have no focal animal ID
Problem
Some pantgrunt records are ad-hoc observations and contain a NULL, empty, or whitespace-only pg_fol_b_foc_id. A WATCHES row requires an animal ID, but assigning the unknown individual UNK would incorrectly imply that a focal existed and merely could not be identified. In the local conversion data examined on 2026-08-21, 5,136 rows had no focal value.
Bad data
SELECT *
FROM clean.pantgrunt_event
WHERE NULLIF(BTRIM(pg_fol_b_foc_id), '') IS NULL
ORDER BY pg_date,
pg_time,
pg_b_actor_id,
pg_b_recipient_id;
Solution
For WATCHES resolution only, normalize an absent pantgrunt focal to the established no-focal AnimID NONE. Create or reuse an Other watch for NONE and the pantgrunt date. Leave the original blank focal value unchanged in clean.pantgrunt_event.
* (#130) PANTGRUNT_EVENT focal/date keys contain conflicting communities
Status: Partially resolved; unresolved for twelve no-focal/date keys. The current loader exclusion is broader than necessary.
Problem
Every pantgrunt is related through EVENTS to one supporting WATCHES row. WATCHES records one CommID, and only one watch of a given Type may exist for a particular Date and AnimID. PANTGRUNTS also records its own CommID. This duplication is intentional: PANTGRUNTS.CommID preserves the community recorded with the pantgrunt in Access, while WATCHES.CommID records the community of the supporting follow or watch period. The two values are permitted to disagree, although the production database reports the disagreement as a warning.
The local conversion data examined on 2026-08-21 contain 15 normalized focal/date keys with both KK and MT pantgrunt communities. These keys cover 104 pantgrunt rows. They fall into two materially different categories:
- Three named-focal keys, covering 48 pantgrunt rows, already have a corresponding follow. The follow supplies an unambiguous B-record WATCHES row whose community is KK. These rows can be converted without choosing or discarding a pantgrunt community: reuse the existing watch and preserve each source value in PANTGRUNTS.CommID. Pantgrunts recorded as MT will appropriately produce community-mismatch warnings.
- Twelve keys, covering 56 pantgrunt rows, have no focal animal ID and no corresponding follow. The conversion normalizes the absent focal to NONE for watch resolution. Each date contains both KK and MT pantgrunts, but the conversion can create only one Other watch for the combination of NONE and that date. There is no source-supported way to decide whether that WATCHES row should contain KK or MT.
The community conflict therefore does not itself prevent a PANTGRUNTS row from storing its source community. The unresolved problem is selecting the canonical WATCHES.CommID when a supporting watch must be created and there is no follow or focal context from which to determine it.
Bad data
The following query shows every conflicting key, any community supplied by a corresponding follow, the pantgrunt communities, and the number of affected pantgrunt rows:
WITH conflict_keys AS (
SELECT COALESCE(NULLIF(BTRIM(pg_fol_b_foc_id), ''), 'NONE') AS watch_animid,
pg_date
FROM clean.pantgrunt_event
GROUP BY COALESCE(NULLIF(BTRIM(pg_fol_b_foc_id), ''), 'NONE'),
pg_date
HAVING COUNT(DISTINCT BTRIM(pg_cl_community_id)) > 1
)
SELECT conflict_keys.watch_animid,
conflict_keys.pg_date,
follow.fol_cl_community_id AS follow_community,
ARRAY_AGG(DISTINCT BTRIM(pantgrunt_event.pg_cl_community_id)
ORDER BY BTRIM(pantgrunt_event.pg_cl_community_id)) AS pantgrunt_communities,
COUNT(*) AS pantgrunt_rows
FROM conflict_keys
JOIN clean.pantgrunt_event
ON COALESCE(NULLIF(BTRIM(pantgrunt_event.pg_fol_b_foc_id), ''), 'NONE') = conflict_keys.watch_animid
AND pantgrunt_event.pg_date = conflict_keys.pg_date
LEFT JOIN clean.follow
ON BTRIM(follow.fol_b_animid) = conflict_keys.watch_animid
AND follow.fol_date = conflict_keys.pg_date
GROUP BY conflict_keys.watch_animid,
conflict_keys.pg_date,
follow.fol_cl_community_id
ORDER BY conflict_keys.pg_date,
conflict_keys.watch_animid;
Examples with named focals
| Focal | Date | Follow community | Pantgrunt communities | Pantgrunt rows |
|---|---|---|---|---|
| FD | 1996-09-08 | KK | KK and MT | 11 |
| ZS | 2013-11-22 | KK | KK and MT | 6 |
| DL | 2013-11-29 | KK | KK and MT | 31 |
For example, the FD records can all use the existing B-record watch for FD on 1996-09-08. That watch remains associated with KK. Each pantgrunt retains either KK or MT in PANTGRUNTS.CommID. The MT records disagree with the watch community, but this is a documented and intentionally preserved source inconsistency rather than a conversion failure.
Examples without a focal
| Watch AnimID used by conversion | Date | Pantgrunt communities | Pantgrunt rows |
|---|---|---|---|
| NONE | 1986-01-11 | KK and MT | 8 |
| NONE | 1986-09-10 | KK and MT | 16 |
| NONE | 1986-10-30 | KK and MT | 3 |
| NONE | 1987-09-26 | KK and MT | 7 |
| NONE | 1992-11-21 | KK and MT | 3 |
| NONE | 1993-12-25 | KK and MT | 2 |
| NONE | 1994-01-19 | KK and MT | 3 |
| NONE | 1994-05-26 | KK and MT | 2 |
| NONE | 1994-07-07 | KK and MT | 2 |
| NONE | 1994-08-01 | KK and MT | 3 |
| NONE | 1994-08-08 | KK and MT | 4 |
| NONE | 1994-10-07 | KK and MT | 3 |
For example, the eight records on 1986-01-11 require a supporting Other watch with AnimID NONE. The WATCHES uniqueness rule permits only one such Other watch on that date. Assigning KK would be unsupported for the MT observations, while assigning MT would be unsupported for the KK observations. Creating two Other watches distinguished only by community is not permitted by the current WATCHES key.
Solution
Refine the conversion so that the three named-focal keys reuse their existing B-record watches and all 48 pantgrunt rows are loaded. Preserve the source community in PANTGRUNTS.CommID and accept the documented warning when it differs from the B-record watch community.
Continue to exclude only the 56 pantgrunt rows belonging to the twelve no-focal/date keys. Report those twelve keys in the sanity check. The investigators must determine whether each date represents one observation period with incorrect community coding, two distinct observation periods that the current WATCHES key cannot represent, or records with incorrect dates or focal values.
Until that refinement is implemented, the current loader conservatively excludes all 15 conflicting keys and therefore excludes 48 otherwise convertible named-focal rows in addition to the 56 genuinely ambiguous no-focal rows.
* (#131) PANTGRUNT_EVENT observer values have no lossless production destination
Status: Unresolved; intentionally retained only in the clean schema.
Problem
The pg_observer field is populated only with JF in the local conversion data examined on 2026-08-21, affecting 1,696 rows across 221 normalized watch keys. PANTGRUNTS has no observer column. FOLLOW_OBSERVERS describes observers assigned to follow periods, so creating FOLLOW_OBSERVERS rows from pantgrunt records would require deciding whether pg_observer represents the same concept and period.
Bad data
SELECT pg_observer,
COUNT(*) AS pantgrunt_rows,
COUNT(DISTINCT (COALESCE(NULLIF(BTRIM(pg_fol_b_foc_id), ''), 'NONE'), pg_date)) AS watch_keys
FROM clean.pantgrunt_event
WHERE pg_observer IS NOT NULL
GROUP BY pg_observer
ORDER BY pg_observer;
Solution
Do not convert pg_observer into FOLLOW_OBSERVERS or another production table without an investigator-approved semantic mapping. Retain the values in clean.pantgrunt_event for later review. The pantgrunt loader explicitly documents that this field is omitted.
* (#132) PANTGRUNT_EVENT participants occur outside their biography intervals
Problem
ROLES requires an event participant's date to fall between BIOGRAPHY_DATA.EntryDate and BIOGRAPHY_DATA.DepartDate, inclusive. Some pantgrunt actors or recipients have a biography row but occur before entry or after departure. In the local conversion data examined on 2026-08-21, 792 rows involved at least one participant outside this interval.
Bad data
SELECT pg.*,
participant.source_column,
participant.animid,
b.b_entrydate,
b.b_departdate,
CASE
WHEN pg.pg_date < b.b_entrydate THEN 'BEFORE ENTRY'
WHEN pg.pg_date > b.b_departdate THEN 'AFTER DEPARTURE'
END AS date_problem
FROM clean.pantgrunt_event AS pg
CROSS JOIN LATERAL (
VALUES ('pg_b_actor_id', pg.pg_b_actor_id),
('pg_b_recipient_id', pg.pg_b_recipient_id)
) AS participant(source_column, animid)
JOIN clean.biography AS b
ON b.b_animid = BTRIM(participant.animid)
WHERE pg.pg_date NOT BETWEEN b.b_entrydate AND b.b_departdate
ORDER BY pg.pg_date,
pg.pg_fol_b_foc_id,
pg.pg_time,
participant.source_column,
participant.animid;
Solution
Temporarily exclude affected rows and report their count in the pantgrunt sanity check. The project investigators must determine whether the pantgrunt date, participant identity, or biography interval should be corrected in Access. Do not weaken the ROLES temporal constraints.
* (#133) PANTGRUNT_EVENT contains invalid nonempty focal animal IDs
Problem
An Other WATCHES row created for a pantgrunt requires a valid BIOGRAPHY_DATA animal ID. Some pg_fol_b_foc_id values are nonempty but do not match a biography animal and therefore cannot be treated as the no-focal value NONE. In the local conversion data examined on 2026-08-21, 70 rows contained such focal values.
Bad data
SELECT pg.*
FROM clean.pantgrunt_event AS pg
WHERE NULLIF(BTRIM(pg.pg_fol_b_foc_id), '') IS NOT NULL
AND NOT EXISTS (
SELECT 1
FROM clean.biography AS b
WHERE b.b_animid = BTRIM(pg.pg_fol_b_foc_id))
ORDER BY pg.pg_date,
pg.pg_fol_b_foc_id,
pg.pg_time,
pg.pg_b_actor_id,
pg.pg_b_recipient_id;
Solution
Temporarily exclude these rows and report their count in the sanity check. The project investigators must identify whether each value is a malformed focal ID, descriptive text, or evidence that no focal existed, then correct Access accordingly. Do not automatically convert a nonempty value to NONE or UNK.
(#134) PANTGRUNT_EVENT notes may be NULL
Problem
EVENTS.Notes is required but permits the empty string. The Access pantgrunt table uses NULL for absent notes. In the local conversion data examined on 2026-08-21, 7,339 rows had NULL pg_notes and no rows contained whitespace-only notes.
Bad data
SELECT *
FROM clean.pantgrunt_event
WHERE pg_notes IS NULL
OR (pg_notes <> '' AND BTRIM(pg_notes) = '')
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time,
pg_b_actor_id,
pg_b_recipient_id;
Solution
At the production load boundary, convert a NULL or whitespace-only pg_notes value to the empty string and otherwise preserve the source text unchanged. Retain the original value in clean.pantgrunt_event.
* (#135) PANTGRUNT_EVENT year values have no lossless production destination
Problem
The production pantgrunt model obtains the event date through WATCHES and has no separate year column. Although pg_year appears redundant with pg_date, the local conversion data examined on 2026-08-21 contain 178 rows where pg_year differs from the year extracted from pg_date. Discarding the field as derived data would therefore lose a recorded discrepancy.
Bad data
records
SELECT *
FROM clean.pantgrunt_event
WHERE pg_year IS NULL
OR pg_year IS DISTINCT FROM extract(year FROM pg_date)::INTEGER
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time,
pg_b_actor_id,
pg_b_recipient_id;
summary
SELECT pg_year,
extract(year FROM pg_date)::INTEGER AS date_year,
pg_year - extract(year FROM pg_date)::INTEGER AS year_difference,
count(*) AS row_count
FROM clean.pantgrunt_event
WHERE pg_year IS DISTINCT FROM
extract(year FROM pg_date)::INTEGER
GROUP BY pg_year,
extract(year FROM pg_date)::INTEGER
ORDER BY pg_year,
date_year;
Solution
Do not load pg_year into another production column or silently replace pg_date. Retain pg_year in clean.pantgrunt_event for investigator review. The pantgrunt loader explicitly documents that the field is omitted pending a project decision about the 178 discrepancies.
(#136) Some GROOM_SCANS direction codes have trailing spaces
Problem
There are 3 GROOM_SCANS records where GS_direction is 'G ' instead of 'G'. The trailing Access padding prevents an exact match with the valid one-character direction code.
Bad Data
Cleaned in the clean schema, so query the easy schema.
SELECT '"' || gs_direction || '"' AS untrimmed_direction,
'"' || BTRIM(gs_direction) || '"' AS trimmed_direction,
count(*) AS row_count
FROM easy.groom_scans
WHERE gs_direction IS DISTINCT FROM BTRIM(gs_direction)
GROUP BY gs_direction,
BTRIM(gs_direction)
ORDER BY gs_direction;
This reports 3 rows having "G ", which normalizes to "G".
Solution
Trim leading and trailing whitespace from GS_direction in the clean schema. This preserves all source rows and allows the direction codes to be mapped normally by the production loader.
Resolved by commit 3ad4f5e1e2355895e5e465c4b8049197c8dce295
* (#137) GROOM_BOUT focal/date keys have no corresponding follow
Problem
A grooming event must relate to a WATCHES row. Normal B-record watches are created from clean.follow, but 1,536 current GROOM_BOUT rows have no follow with the same focal and date after animal-ID whitespace normalization. GROOM_BOUT has no community column, so the loader cannot create a watch directly from the source row.
Of these rows, 1,399 have exactly one COMMUNITY_MEMBERSHIP row covering the observation date. The remaining 137 do not have exactly one dated membership and must not be assigned a community by inference.
Bad data
note: querying clean, the schema read by the production loader
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow
WHERE follow.fol_b_animid = gb.grm_fol_b_focal_animid
AND follow.fol_date = gb.grm_fol_date)
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid;
The temporarily excluded subset is identified by:
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow
WHERE follow.fol_b_animid = gb.grm_fol_b_focal_animid
AND follow.fol_date = gb.grm_fol_date)
AND 1 <> (
SELECT count(*)
FROM clean.community_membership
WHERE community_membership.cm_b_animid =
gb.grm_fol_b_focal_animid
AND gb.grm_fol_date BETWEEN
community_membership.cm_start_date
AND community_membership.cm_end_date)
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid;
Solution
As established for ad-hoc observations in Problem #128, prefer an existing B-record WATCHES row and otherwise reuse an existing Other watch for the focal and date. When no watch exists and exactly one COMMUNITY_MEMBERSHIP interval covers the observation date, create an Other watch using that membership's community. This preserves the grooming bout without falsely representing it as part of a follow.
Do not infer a community from unrelated observations when there is not exactly one dated membership. Temporarily exclude those 137 rows from the production groom-bout loader. Three overlap Problem #138, so this exclusion adds 134 rows to the excluded union. Investigators must determine the focal's community on the observation date or correct the missing follow or membership data in Access.
* (#138) GROOM_BOUT initiator or terminator is not a member of the grooming pair
Problem
After applying the approved Problem #112 interpretation of U as unknown and trimming animal IDs under Problem #139, 2,149 GROOM_BOUT rows have a nonempty initiator or terminator animal ID that is neither the focal nor the recorded grooming partner. There are 1,776 initiator mismatches and 2,133 terminator mismatches, with overlap between those sets.
Production GROOMINGS.Initiator and GROOMINGS.Terminator values must reference ROLES.PID rows belonging to participants in the same grooming event. The source does not establish whether the mismatched value, the recorded partner, or another field is incorrect, so the loader cannot safely choose a role or add another participant.
Bad data
note: querying clean, the schema read by the production loader
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
gb.grm_direction,
invalid_reference.source_column,
invalid_reference.animid AS offending_animid,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
CROSS JOIN LATERAL (
VALUES ('GRM_B_initiator_AnimId', gb.grm_b_initiator_animid),
('GRM_B_terminator_AnimId', gb.grm_b_terminator_animid)
) AS invalid_reference(source_column, animid)
WHERE invalid_reference.animid IS NOT NULL
AND invalid_reference.animid NOT IN (
gb.grm_fol_b_focal_animid,
gb.grm_b_partner_animid)
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
invalid_reference.source_column;
Solution
Temporarily exclude affected rows from the production groom-bout loader. The project investigators must determine the correct grooming partner and the correct initiator or terminator in Access. Keep the production requirement that an initiator or terminator reference a participant role in the same grooming event.
(#139) GROOM_BOUT animal IDs contain edge whitespace
Problem
There are 50 animal-ID values with leading or trailing whitespace across 49 GROOM_BOUT rows: 36 focal values, 9 partner values, 2 initiator values, and 3 terminator values. The first runtime failure used partner value BE , which does not reference BIOGRAPHY_DATA even though the trimmed value BE does.
All 36 focal and all 9 partner values identify existing BIOGRAPHY rows after trimming. Trimming is the established lossless conversion treatment for animal-ID edge whitespace.
Bad data
note: querying easy because the values are normalized in clean
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
animal_id.source_column,
animal_id.animid,
BTRIM(animal_id.animid) AS trimmed_animid
FROM easy.groom_bout AS gb
CROSS JOIN LATERAL (
VALUES ('GRM_FOL_B_focal_AnimId', gb.grm_fol_b_focal_animid),
('GRM_B_partner_AnimId', gb.grm_b_partner_animid),
('GRM_B_initiator_AnimId', gb.grm_b_initiator_animid),
('GRM_B_terminator_AnimId', gb.grm_b_terminator_animid)
) AS animal_id(source_column, animid)
WHERE animal_id.animid IS DISTINCT FROM BTRIM(animal_id.animid)
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
animal_id.source_column;
Solution
Trim leading and trailing whitespace from the four GROOM_BOUT animal-ID columns in the clean schema. Apply the Problem #112 conversion of initiator or terminator U values to SQL NULL after trimming. Do not change GRM_direction.
* (#140) GROOM_BOUT focal or partner animal IDs are absent from BIOGRAPHY
Problem
After the lossless Problem #139 whitespace normalization, 412 GROOM_BOUT rows have a focal or grooming partner that is absent from BIOGRAPHY: 131 rows have an absent focal and 337 have an absent partner. Some rows occur in both counts. Production WATCHES.AnimID and ROLES.Participant values must reference BIOGRAPHY_DATA.AnimID.
The first otherwise eligible runtime failure is the 1992-10-13 PF/MGE bout. The source does not identify a production animal for MGE, so no sentinel or other animal can be substituted safely.
Bad data
note: querying clean, the schema read by the production loader
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
missing_participant.source_column,
missing_participant.animid,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
CROSS JOIN LATERAL (
VALUES ('GRM_FOL_B_focal_AnimId', gb.grm_fol_b_focal_animid),
('GRM_B_partner_AnimId', gb.grm_b_partner_animid)
) AS missing_participant(source_column, animid)
WHERE NOT EXISTS (
SELECT 1
FROM clean.biography
WHERE biography.b_animid = missing_participant.animid)
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
missing_participant.source_column;
Solution
Temporarily exclude affected rows from the production groom-bout loader. Of the 412 matches, 206 overlap Problems #137 or #138 and 206 are newly excluded. Investigators must identify the intended animals and correct the Access data. Keep the production foreign keys to BIOGRAPHY_DATA intact.
* (#141) GROOM_BOUT participants are observed before entry into the study
Problem
Seven GROOM_BOUT rows involve a focal or partner before that animal's BIOGRAPHY.EntryDate: one focal occurrence and six partner occurrences. Production rejects a role dated before its participant entered the study. The first otherwise eligible failure observes SN on 1996-03-11, before its 1996-06-03 entry date.
The identifying predicate uses a strict less-than comparison, so an observation on EntryDate remains valid.
Bad data
note: querying clean, the schema read by the production loader
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
early_participant.source_column,
early_participant.animid,
biography.b_entrydate,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
CROSS JOIN LATERAL (
VALUES ('GRM_FOL_B_focal_AnimId', gb.grm_fol_b_focal_animid),
('GRM_B_partner_AnimId', gb.grm_b_partner_animid)
) AS early_participant(source_column, animid)
JOIN clean.biography
ON biography.b_animid = early_participant.animid
WHERE gb.grm_fol_date < biography.b_entrydate
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
early_participant.source_column;
Solution
Temporarily exclude affected rows from the production groom-bout loader. One overlaps Problems #137, #138, or #140, so this exclusion adds six rows to the excluded union. Investigators must correct the observation date, participant, or biography entry date in Access. Keep the production rule and its inclusive EntryDate boundary intact.
* (#142) GROOM_BOUT participants are observed after departure from the study
Problem
Sixty GROOM_BOUT rows involve a focal or partner after that animal's BIOGRAPHY.DepartDate: five rows have an affected focal and 56 have an affected partner. Some rows occur in both counts. Production rejects a role dated after its participant departed the study. The first otherwise eligible failure observes FI on 1997-07-30, one day after its 1997-07-29 departure date.
The identifying predicate uses a strict greater-than comparison, so an observation on DepartDate remains valid.
Bad data
note: querying clean, the schema read by the production loader
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
late_participant.source_column,
late_participant.animid,
biography.b_departdate,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
CROSS JOIN LATERAL (
VALUES ('GRM_FOL_B_focal_AnimId', gb.grm_fol_b_focal_animid),
('GRM_B_partner_AnimId', gb.grm_b_partner_animid)
) AS late_participant(source_column, animid)
JOIN clean.biography
ON biography.b_animid = late_participant.animid
WHERE gb.grm_fol_date > biography.b_departdate
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
late_participant.source_column;
Solution
Temporarily exclude affected rows from the production groom-bout loader. Eleven overlap Problems #137, #138, #140, or #141, so this exclusion adds 49 rows to the excluded union. Investigators must correct the observation date, participant, or biography departure date in Access. Keep the production rule and its inclusive DepartDate boundary intact.
* (#143) GROOM_BOUT other-partner flags have non-boolean values
Problem
GROOMINGS.Others is a required boolean, while GROOM_BOUT.GRM_other_partners contains several encodings. The established Access boolean-style convention treats NULL or the empty string as false, and the source also contains lowercase y and n values that differ only by case. There are 4,266 blank or NULL values and 561 lowercase values.
After those lossless boundary normalizations, nine rows retain values that cannot be mapped to a boolean: B in three rows, G in one row, M in two rows, and U in three rows. The first otherwise eligible runtime failure has U on the 2002-07-28 SA/SR bout.
Bad data
summary before clean-schema normalization
SELECT COALESCE(grm_other_partners, '<NULL>') AS raw_value,
count(*) AS row_count
FROM easy.groom_bout
GROUP BY grm_other_partners
ORDER BY raw_value;
unmappable values in clean, the schema read by the production loader
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
gb.grm_other_partners,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
WHERE gb.grm_other_partners NOT IN ('Y', 'N')
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid;
Solution
During construction of the clean schema, convert NULL and empty GRM_other_partners values to N and normalize lowercase y and n to uppercase. Temporarily exclude the nine rows whose normalized value is not Y or N. One overlaps Problems #137, #138, #140, #141, or #142, so this exclusion adds eight rows to the excluded union.
Investigators must determine whether B, G, M, and U mean that other grooming partners were present and correct the Access data. Keep GROOMINGS.Others required and boolean.
* (#144) GROOM_BOUT times are outside the production observation window
Problem
Six GROOM_BOUT rows have start and stop times outside the production EVENTS range of 04:00 through 20:00 inclusive. All six violate both the start and stop constraints. The first runtime failure is the 2007-03-28 NUR/KS bout from 02:55 through 02:57.
The source does not establish whether these are valid nighttime observations or mistyped times. Changing the production observation window requires a separate decision.
Bad data
note: querying clean, the schema read by the production loader
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_time_end,
gb.grm_b_partner_animid,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
WHERE gb.grm_time_begin < '04:00'::TIME
OR gb.grm_time_begin > '20:00'::TIME
OR (gb.grm_time_end IS NOT NULL
AND (gb.grm_time_end < '04:00'::TIME
OR gb.grm_time_end > '20:00'::TIME))
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid;
Solution
Temporarily exclude the six rows from the production groom-bout loader. None overlap Problems #137, #138, #140, #141, #142, or #143. Investigators must confirm the intended times or decide separately whether the production observation window should change. Keep the current production constraints intact.
* (#145) GROOM_BOUT records the same animal as focal and partner
Problem
Four GROOM_BOUT rows identify the same animal as both the focal and grooming partner. Production represents grooming as a dyadic event and requires each ROLES participant to be unique within an event, so the second role violates the unique participant-and-event constraint. The first failing row is the 2011-07-31 TOM/TOM bout, whose source problems text includes "WRONG FOCAL".
Bad data
note: querying clean, the schema read by the production loader
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
WHERE gb.grm_fol_b_focal_animid = gb.grm_b_partner_animid
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid;
Solution
Temporarily exclude the four rows from the production groom-bout loader. None overlap Problems #137, #138, #140, #141, #142, #143, or #144. Investigators must identify the intended focal or grooming partner and correct the Access data. Keep the production requirement that a participant occur at most once in an event.
* (#146) GROOM_SCANS participants are observed after departure from the study
Problem
Sixty-four GROOM_SCANS rows involve a grooming participant after that animal's BIOGRAPHY.DepartDate: eight rows have an affected GS_B_chimp1_AnimId and 56 have an affected GS_B_chimp2_AnimId. Production rejects a role dated after its participant departed the study. The first runtime failure observes MT on 1978-05-23, after its 1974-11-01 departure date.
The identifying predicate uses a strict greater-than comparison, so an observation on DepartDate remains valid. Seventeen participant occurrences are on their departure date.
Bad data
note: querying clean, the schema read by the production loader
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
late_participant.source_column,
late_participant.animid,
biography.b_departdate,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
CROSS JOIN LATERAL (
VALUES ('GS_B_chimp1_AnimId', gs.gs_b_chimp1_animid),
('GS_B_chimp2_AnimId', gs.gs_b_chimp2_animid)
) AS late_participant(source_column, animid)
JOIN clean.biography
ON biography.b_animid = late_participant.animid
WHERE gs.gs_date > biography.b_departdate
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
late_participant.source_column;
Solution
Temporarily exclude the 64 affected rows from the production B-record groom-scan loader. Investigators must correct the observation date, participant, or biography departure date in Access. Keep the production rule and its inclusive DepartDate boundary intact.
* (#147) GROOM_SCANS focal/date keys have no corresponding follow
Problem
A B-record groom scan must relate to a B-type WATCHES row. There are 673 current GROOM_SCANS rows whose focal and date have no matching clean.follow row. Of these, 665 rows have exactly one COMMUNITY_MEMBERSHIP interval covering the observation date and eight rows, comprising three focal/date keys, have no dated membership.
The first runtime failure is the 1978-06-29 ST scan at 14:15. Its source table identifies it as B-record data, and ST has exactly one community membership on that date, so the community is unambiguous even though the corresponding follow is absent.
Bad data
note: querying clean, the schema read by the production loader
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow
WHERE follow.fol_b_animid = gs.gs_fol_b_focal_animid
AND follow.fol_date = gs.gs_date)
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
The temporarily excluded subset is identified by:
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow
WHERE follow.fol_b_animid = gs.gs_fol_b_focal_animid
AND follow.fol_date = gs.gs_date)
AND 1 <> (
SELECT count(*)
FROM clean.community_membership
WHERE community_membership.cm_b_animid =
gs.gs_fol_b_focal_animid
AND gs.gs_date BETWEEN
community_membership.cm_start_date
AND community_membership.cm_end_date)
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
Solution
Reuse an existing B-type WATCHES row when one was created by an earlier loader. Otherwise, when exactly one COMMUNITY_MEMBERSHIP interval covers the observation date, create a B watch using that membership's community. The GROOM_SCANS source identifies these observations as B-record data, so this does not invent the watch type.
Do not infer a community when no dated membership exists. Temporarily exclude the eight unresolved rows; they do not overlap Problem #146. Investigators must determine the focal's community on the observation date or correct the missing follow or membership data in Access.
* (#148) GROOM_SCANS participants are observed before entry into the study
Problem
Four GROOM_SCANS rows involve a grooming participant before that animal's BIOGRAPHY.EntryDate. All four affected values are in GS_B_chimp2_AnimId. Production rejects a role dated before its participant entered the study. The first runtime failure observes KP on 1981-09-15, before its 1997-03-16 entry date.
The identifying predicate uses a strict less-than comparison, so an observation on EntryDate remains valid. No current GROOM_SCANS participant is observed exactly on its entry date.
Bad data
note: querying clean, the schema read by the production loader
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
early_participant.source_column,
early_participant.animid,
biography.b_entrydate,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
CROSS JOIN LATERAL (
VALUES ('GS_B_chimp1_AnimId', gs.gs_b_chimp1_animid),
('GS_B_chimp2_AnimId', gs.gs_b_chimp2_animid)
) AS early_participant(source_column, animid)
JOIN clean.biography
ON biography.b_animid = early_participant.animid
WHERE gs.gs_date < biography.b_entrydate
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
early_participant.source_column;
Solution
Temporarily exclude the four affected rows from the production B-record groom-scan loader. They do not overlap Problems #146 or #147. Investigators must correct the observation date, participant, or biography entry date in Access. Keep the production rule and its inclusive EntryDate boundary intact.
* (#149) GROOM_SCANS times are outside the production observation window
Problem
Sixty-one GROOM_SCANS rows have times outside the production EVENTS range of 04:00 through 20:00 inclusive. The values range from 00:00 through 23:50. The first runtime failure is the 1982-01-13 PI scan at 03:30.
The source does not establish whether these are valid nighttime observations or mistyped times. Changing the production observation window requires a separate decision. No current GROOM_SCANS row occurs exactly at 04:00 or 20:00.
Bad data
note: querying clean, the schema read by the production loader
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE gs.gs_time < '04:00'::TIME
OR gs.gs_time > '20:00'::TIME
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
Solution
Temporarily exclude the 61 affected rows from the production B-record groom-scan loader. They do not overlap Problems #146, #147, or #148. Investigators must confirm the intended times or decide separately whether the production observation window should change. Keep the current production constraints intact.
* (#150) GROOM_SCANS unknown direction cannot use the production UNKPair role
Problem
Thirty-one GROOM_SCANS rows have GS_direction = 'U'. The loader maps this source code to paired UNKPair roles, which represent a dyadic interaction whose direction is unknown. The current production roles_func and EVENTS documentation permit Actor, Actee, and Mutual roles for GSCAN events but reject UNKPair.
The first runtime failure is the 1982-02-07 ST scan at 11:55 with participants FF and GB. Its source comment says "NO DIRECTION INDICATED ON TIKI OR ON B RECORDS". None of the 31 rows overlaps Problems #146 through #149.
Bad data
note: querying clean, the schema read by the production loader
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE gs.gs_direction = 'U'
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
Solution
Temporarily exclude the 31 source rows while retaining the current production role restriction. The investigator does not want structural schema changes in the B-record loader commit. Flag permitting paired UNKPair roles for GSCAN events, with the corresponding EVENTS documentation change, as a structural problem for the next iteration.
Do not convert U to Mutual because unknown direction does not establish symmetric grooming.
* (#151) GROOM_SCANS participants are absent from BIOGRAPHY
Problem
Two hundred twenty-five GROOM_SCANS rows have a grooming participant absent from BIOGRAPHY: 33 affected values are in GS_B_chimp1_AnimId and 192 are in GS_B_chimp2_AnimId. Production ROLES.Participant must reference BIOGRAPHY_DATA.AnimID. The first runtime failure is the 1982-03-09 FF scan at 15:45, whose second participant is TP.
The source does not identify a production animal that can safely replace an absent participant.
Bad data
note: querying clean, the schema read by the production loader
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
missing_participant.source_column,
missing_participant.animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
CROSS JOIN LATERAL (
VALUES ('GS_B_chimp1_AnimId', gs.gs_b_chimp1_animid),
('GS_B_chimp2_AnimId', gs.gs_b_chimp2_animid)
) AS missing_participant(source_column, animid)
WHERE NOT EXISTS (
SELECT 1
FROM clean.biography
WHERE biography.b_animid = missing_participant.animid)
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
missing_participant.source_column;
Solution
Temporarily exclude the 225 affected rows from the production B-record groom-scan loader. They do not overlap Problems #146 through #150. Investigators must identify the intended animals and correct the Access data. Keep the production foreign key to BIOGRAPHY_DATA intact.
* (#152) GROOM_SCANS direction values cannot be mapped to production roles
Problem
After the Problem #136 whitespace normalization, 48 GROOM_SCANS rows have a direction other than G, R, M, or U: 43 are NULL, one is F, one is H, and three are T. The first runtime failure that reached the direction mapping has H on the 1989-06-22 GB scan at 08:15.
The source does not establish which participant groomed the other, whether grooming was mutual, or whether direction was unknown. The loader cannot choose production roles safely.
Bad data
note: querying clean, the schema read by the production loader
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE COALESCE(gs.gs_direction, '') NOT IN ('G', 'R', 'M', 'U')
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
Solution
Temporarily exclude the 48 affected rows from the production B-record groom-scan loader. Three overlap Problems #146 through #151, so this exclusion adds 45 rows to the excluded union. Investigators must determine the intended direction and correct the Access data. Do not infer roles from participant order or map missing direction to Mutual or unknown without a documented decision.
* (#153) GROOM_SCANS records the same animal as both participants
Problem
Twelve GROOM_SCANS rows identify the same animal in GS_B_chimp1_AnimId and GS_B_chimp2_AnimId. Production represents groom scans as dyadic events and requires each ROLES participant to be unique within an event. The first runtime failure is the 2015-03-16 NUR/NUR scan at 13:00, whose source comment says "Same ID for chimp1 and chimp2".
Bad data
note: querying clean, the schema read by the production loader
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE gs.gs_b_chimp1_animid = gs.gs_b_chimp2_animid
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
Solution
Temporarily exclude the 12 affected rows from the production B-record groom-scan loader. They do not overlap Problems #146 through #152. Investigators must identify the intended first or second participant, or decide separately how self-grooming should be represented. Keep the production requirement that a participant occur at most once in an event.
* (#154) GROOM_SCANS records have no observation time
Problem
Eight GROOM_SCANS rows have NULL GS_time values. Production EVENTS requires a time, and all eight source comments say "no time". None of the rows overlaps Problems #146 through #153.
Bad data
note: querying clean, the schema read by the production loader
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE gs.gs_time IS NULL
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
Solution
Temporarily exclude the eight affected rows from the production B-record groom-scan loader. Investigators must recover the observation times from another source or document a separate policy for missing event times. Do not invent a time from the scan date, record order, or neighboring observations.
(#155) FERTILITY boundary-type codes need production support rows
Problem
The 168 clean.fertility rows use four StartType values and four StopType
values. Production FERTILITY requires these values to reference
FERTILITY_STARTS and FERTILITY_STOPS, but both support tables are empty.
The fertility domains exactly match the BIOGRAPHY_DATA boundary-type domains:
B, C, I, and O match ENTRYTYPES, while D, E, O, and P match
DEPARTTYPES.
Bad data
note: querying clean, the schema read by the production loader
SELECT 'StartType' AS source_column,
fertility.starttype AS code,
COUNT(*) AS rows
FROM clean.fertility AS fertility
GROUP BY fertility.starttype
UNION ALL
SELECT 'StopType',
fertility.stoptype,
COUNT(*)
FROM clean.fertility AS fertility
GROUP BY fertility.stoptype
ORDER BY source_column, code;
The refreshed 2026-09-13 source contains StartType B (70 rows), C (56),
I (34), and O (8), and StopType D (74), E (1), O (75), and P
(18).
Solution
The project approved using the corresponding ENTRYTYPES and DEPARTTYPES
meanings for fertility boundary types. Populate FERTILITY_STARTS with B
(Birth), C (Start of confirmed AnimID), I (Immigration), and O
(Initiation of close observation). Populate FERTILITY_STOPS with D (Death),
E (Emigration), O (End of observation; Present in the most recent census),
and P (Permanent disappearance).
This is a support-table decision and excludes no source rows.
(#156) FERTILITY source intervals require ordered dates
Problem
Production FERTILITY does not have a table constraint requiring StartDate <=
StopDate. Loading a reversed source interval would therefore preserve an
invalid period without necessarily producing a database error.
Bad data
note: querying clean, the schema read by the production loader
SELECT fertility.studyid,
fertility.animalid,
fertility.startdate,
fertility.starttype,
fertility.stopdate,
fertility.stoptype
FROM clean.fertility AS fertility
WHERE fertility.startdate > fertility.stopdate
ORDER BY fertility.studyid,
fertility.animalid,
fertility.startdate,
fertility.stopdate;
The refreshed 2026-09-13 source contains zero reversed intervals.
Solution
Enforce StartDate <= StopDate in the read-only fertility conversion sanity
gate. Do not add a production table constraint as part of this conversion.
Abort before writing any production rows if a future source snapshot contains
a reversed interval.
This decision excludes no source rows.
* (#157) Selected FERTILITY periods cross BIOGRAPHY_DATA boundaries
Problem
Nine clean.fertility rows extend beyond one or more BIOGRAPHY_DATA date
boundaries. Production does not require fertility periods to remain within
BirthDate, EntryDate, and DepartDate, so the database does not establish
whether these differences are valid history or source errors.
Investigators reviewed the nine rows and selected six for temporary exclusion: AR, BH, GLIB1, MAM, NV, and RAF. The remaining boundary differences for SAF, SHO, and SN1 are approved for conversion and are not excluded.
Bad data
note: querying clean, the schema read by the production loader; all six source fields identify each approved exclusion
SELECT fertility.studyid,
fertility.animalid,
fertility.startdate,
fertility.starttype,
fertility.stopdate,
fertility.stoptype
FROM clean.fertility AS fertility
WHERE (fertility.studyid,
fertility.animalid,
fertility.startdate,
fertility.starttype,
fertility.stopdate,
fertility.stoptype)
IN ((4, 'AR', DATE '1985-07-11', 'B', DATE '1987-07-19', 'D'),
(4, 'BH', DATE '1971-07-05', 'B', DATE '1971-09-20', 'D'),
(4, 'GLIB1', DATE '2011-07-11', 'B', DATE '2012-01-21', 'D'),
(4, 'MAM', DATE '2004-02-12', 'B', DATE '2011-12-07', 'D'),
(4, 'NV', DATE '1965-09-15', 'I', DATE '1975-03-12', 'D'),
(4, 'RAF', DATE '1986-10-21', 'C', DATE '1996-03-27', 'D'))
ORDER BY fertility.animalid,
fertility.startdate;
The query returns exactly six rows. Because this is a new FERTILITY loader, none overlaps an earlier source-row exclusion. The distinct exclusion union is six rows, leaving 162 eligible rows from the refreshed 168-row source.
Solution
Temporarily exclude the six identified rows from the production FERTILITY loader. Keep the full six-column predicate adjacent to its diagnostic query so the exclusion cannot broaden silently. Investigators must correct the Access fertility or biography dates, or approve a later interpretation that permits these periods, before the rows are restored.
Do not exclude SAF, SHO, or SN1. Their boundary differences were reviewed and approved for conversion.
(#158) FERTILITY StudyId 4 has no production STUDIES row
Problem
All 168 clean.fertility rows have integer StudyId 4. Production
FERTILITY.Study is text and must reference STUDIES, but the follow-derived
support data contains no Study code 4. The Access snapshot contains no study
lookup table, description, foreign key, or imported metadata that gives StudyId
4 a textual name.
Bad data
note: querying clean, the schema read by the production loader
SELECT fertility.studyid,
COUNT(*) AS rows
FROM clean.fertility AS fertility
GROUP BY fertility.studyid
ORDER BY fertility.studyid;
The refreshed 2026-09-13 source returns only StudyId 4, with 168 rows.
Solution
Preserve the source value without inventing unsupported semantics. Add Study
code 4 to STUDIES with description 4, and map clean.fertility.studyid to
FERTILITY.Study using studyid::text at the load boundary.
This support and representation decision excludes no source rows.
(#159) FOOD_LOOKUP status flags need explicit conversion semantics
Problem
clean.food_lookup contains FL_exclude and
FL_unverified flags, but the mung support and event loaders ignored
both. Loading all lookup rows would admit names explicitly marked for
exclusion; treating unverified names the same way would discard data without
investigator approval.
Bad data
The following is a dated diagnostic. Rerun it against the refreshed clean schema and record exact referring rows before implementing exclusions.
WITH normalized_lookup AS (
SELECT UPPER(BTRIM(fl_local_food_name)) AS foodname,
fl_exclude,
"FL_unverified" AS fl_unverified
FROM clean.food_lookup
),
used AS (
SELECT 'seq1' AS slot,
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
UPPER(BTRIM(fb_fl_local_food_name)) AS foodname
FROM clean.food_bout
UNION ALL
SELECT 'seq2',
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
UPPER(BTRIM(fb_local_food_name2))
FROM clean.food_bout
WHERE NULLIF(BTRIM(fb_local_food_name2), '') IS NOT NULL
)
SELECT used.*,
normalized_lookup.fl_exclude,
normalized_lookup.fl_unverified
FROM used
JOIN normalized_lookup USING (foodname)
WHERE normalized_lookup.fl_exclude IS TRUE
OR normalized_lookup.fl_unverified IS TRUE
ORDER BY used.fb_fol_date,
used.fb_fol_b_focal_animid,
used.fb_begin_feed_time,
used.slot;
The 2026-09-13 snapshot had 18 excluded lookup rows and 359 unverified lookup rows. Excluded names were referenced by 538 primary and 24 secondary bout items, 562 occurrences total. Unverified names were referenced by 728 bout items. These sets may overlap and must be profiled separately.
Solution
FL_exclude = true disqualifies the lookup entry. Do not load that
name into FOOD_NAMES, and exactly exclude every referring food-bout
item after approved case and edge-whitespace normalization.
FL_unverified = true does not disqualify a lookup entry or
referring bout. Preserve the flag in clean, document that it has no
production destination, and load the otherwise eligible name and bout.
The sanity gate must prove that no excluded name enters production and that no unverified name is omitted merely because it is unverified.
(#160) Converted food events require a consumer ROLES row
Problem
The mung food loader inserted EVENTS and
FOOD_EVENTS rows but inserted no ROLES row. Production
documentation says the food event's role identifies the individual consuming
the food. The roles trigger permits at most one role and requires that
participant to equal the related watch focal. Although absence currently
produces a warning rather than a hard constraint failure, omitting the consumer
would make the conversion incomplete.
Bad data
Run after a rollback-only or disposable-database load.
SELECT events.eid,
watches.wid,
watches.animid AS focal,
COUNT(roles.pid) AS role_rows,
MIN(roles.role) AS role,
MIN(roles.participant) AS participant
FROM sokwedb.events
JOIN sokwedb.watches
ON watches.wid = events.wid
LEFT JOIN sokwedb.roles
ON roles.eid = events.eid
WHERE events.behavior = 'FOOD'
GROUP BY events.eid,
watches.wid,
watches.animid
HAVING COUNT(roles.pid) <> 1
OR MIN(roles.role) <> 'Solo'
OR MIN(roles.participant) <> watches.animid
ORDER BY events.eid;
Solution
For every eligible food bout, insert exactly one ROLES row related
to the newly created event. Use role code Solo and the selected B
watch's exact AnimID as Participant. Capture the generated EID;
do not infer event identity from sequence state or source ordering.
This decision excludes no source rows. The loader and parity checks must prove one food event, one Solo consumer role, and one or two contiguous food-detail rows per eligible source bout.
* (#161) FOOD_VARIATIONS conversion is deferred
Problem
clean.food_variations_lookup has a corresponding production
housekeeping.FOOD_VARIATIONS table, but the mung branch never
loaded it. The current food-event conversion requires decisions for missing
and unresolved LocalName values that are independent of loading food bouts and
their two support tables.
Bad data
This is a dated scope diagnostic, not an exclusion query.
SELECT COUNT(*) AS source_rows,
COUNT(*) FILTER (
WHERE NULLIF(BTRIM(fvl_food_spelling_variant), '') IS NULL
) AS missing_variants,
COUNT(*) FILTER (
WHERE NULLIF(BTRIM(fvl_fl_local_food_name), '') IS NULL
) AS missing_local_names
FROM clean.food_variations_lookup;
The 2026-09-13 snapshot contained 1,575 rows, no blank variants, 224 blank or
NULL local names, and 38 additional nonblank local names without a normalized
match in clean.food_lookup. Rerun and extend the query in the
future variations conversion; do not reuse these counts as exclusions.
Solution
Explicitly defer clean.food_variations_lookup to a separate future
conversion. The present food conversion must not insert
FOOD_VARIATIONS rows, must not depend on that table to normalize
bout names, and must not claim variation parity.
Preserve this issue and the source table for the future conversion. The deferral excludes no food-bout rows and does not authorize dropping variation data.
(#162) Eligible FOOD_LOOKUP rows lack generalized scientific names
Problem
The approved FOOD_NAMES.Description source is
fl_sci_food_name_gen, but 360 eligible lookup rows have a blank or
NULL value. All 360 names are used by food bouts, in 812 primary and 111
secondary items. Of these names, 359 are marked FL_unverified and
must not be disqualified under Problem #159. The target description is NOT
NULL, nonempty, and case-insensitively unique.
Solution
Approved by the investigator on 2026-09-13: use
Unknown -- <local name> for an eligible blank generalized name.
This stable value identifies the exact source name and does not consult
fl_sci_food_name. Nonblank descriptions continue to follow
Problem #87. This mapping excludes no rows.
(#163) FOOD_BOUT duration disagrees with elapsed event time
Problem
The refreshed source has 1,136 rows where fb_duration differs from
the exact minutes between begin and end. Production stores the original begin
and end times but has no independent duration column.
Solution
Do not alter event times to reproduce fb_duration. Retain the
field in clean.food_bout, emit a sanity warning with the refreshed
count, and document that it has no production destination. This issue causes
no exclusions.
(#164) FOOD_BOUT update metadata has no production destination
Problem
The refreshed source has 17,702 rows with non-NULL fb_update.
None of the food target tables has a corresponding update-metadata column.
Solution
Retain fb_update in clean.food_bout, emit a sanity
warning with the refreshed count, and do not overload event notes or another
production field. This issue causes no exclusions.
* (#165) ATTENDANCE contains exact duplicate rows
Problem
The source table has no primary key and contains values that are identical in all 17 columns. Loading every copy would create indistinguishable production events and repeated sequence values. Retaining an arbitrary copy would not be an exact source-row policy because no source value distinguishes the copies.
Bad data
Run against the refreshed clean schema. The grouping explicitly
compares all 17 typed source columns.
SELECT attendance.*,
COUNT(*) AS copies
FROM clean.attendance AS attendance
GROUP BY attendance.a_date,
attendance.a_cl_community_id,
attendance.a_b_animid,
attendance.a_seq_num,
attendance.a_type_of_cycle,
attendance.a_observer1_id,
attendance.a_observer2_id,
attendance.a_bananas_given,
attendance.a_degree_of_arrival,
attendance.a_degree_of_departure,
attendance.a_time_start,
attendance.a_time_end,
attendance.a_duration_of_obs,
attendance.day,
attendance.mo,
attendance.yr,
attendance.cycle_old
HAVING COUNT(*) > 1
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num,
attendance.a_time_start;
The 2026-09-14 snapshot contained 207 duplicate values, each occurring twice: 414 source rows total.
Solution
Approved by the investigator on 2026-09-14: exclude every copy of each exact
duplicate value. Do not use DISTINCT to retain one arbitrary copy.
The shared attendance projection and loader must include this query beside the
Problem #165 predicate and prove that every row whose complete typed value has
multiplicity greater than one is excluded.
Implemented in conversion/attendance_projection.sql as
duplicate_count > 1, partitioned over all 17 columns. On
2026-09-15 the investigator approved semantic validation: the dated 207 values
and 414 rows are audit evidence, not frozen acceptance criteria. Sanity
asserts that every refreshed duplicate copy is classified and excluded.
Remove only when exact duplicates leave the source.
* (#166) ATTENDANCE sequence values are not contiguous per animal and date
Problem
Production attendance sequence values are expected to be exactly
1..N for each animal/date. Exact duplicates and other targeted
row exclusions can themselves create duplicate values or gaps, so sequence
validity must be checked after all other attendance exclusions.
Bad data
This query expresses the approved staging against the 2026-09-14 schema. Keep it synchronized with Problems #165 and #168 through #178 when predicates change.
WITH source AS (
SELECT row_number() OVER () AS source_id,
attendance.*,
COUNT(*) OVER (
PARTITION BY attendance.a_date,
attendance.a_cl_community_id,
attendance.a_b_animid,
attendance.a_seq_num,
attendance.a_type_of_cycle,
attendance.a_observer1_id,
attendance.a_observer2_id,
attendance.a_bananas_given,
attendance.a_degree_of_arrival,
attendance.a_degree_of_departure,
attendance.a_time_start,
attendance.a_time_end,
attendance.a_duration_of_obs,
attendance.day,
attendance.mo,
attendance.yr,
attendance.cycle_old
) AS duplicate_count
FROM clean.attendance AS attendance
),
independent_eligible AS (
SELECT source.*
FROM source
JOIN sokwedb.biography_data AS biography
ON biography.animid = source.a_b_animid::text
WHERE source.duplicate_count = 1
AND source.a_date BETWEEN biography.entrydate AND biography.departdate
AND COALESCE(BTRIM(source.a_observer1_id::text), 'NONE')
<> COALESCE(BTRIM(source.a_observer2_id::text), 'NONE')
AND BTRIM(source.a_type_of_cycle::text)
NOT IN ('-', '0.30', '0.33')
AND source.cycle_old IS NOT NULL
AND source.cycle_old::text = BTRIM(source.cycle_old::text)
AND (source.a_bananas_given IS NULL
OR source.a_bananas_given = TRUNC(source.a_bananas_given))
AND source.a_time_start IS NOT NULL
AND source.a_time_end IS NOT NULL
AND source.a_time_start <= source.a_time_end
AND source.a_duration_of_obs IS NOT DISTINCT FROM
EXTRACT(EPOCH FROM
(source.a_time_end - source.a_time_start)) / 60
AND source.day IS NOT DISTINCT FROM
EXTRACT(DAY FROM source.a_date)::integer
AND source.mo IS NOT DISTINCT FROM
EXTRACT(MONTH FROM source.a_date)::integer
AND source.yr IS NOT DISTINCT FROM
EXTRACT(YEAR FROM source.a_date)::integer
),
overlap_rows AS (
SELECT DISTINCT first_row.source_id
FROM independent_eligible AS first_row
JOIN independent_eligible AS second_row
ON second_row.source_id <> first_row.source_id
AND second_row.a_date = first_row.a_date
AND second_row.a_b_animid = first_row.a_b_animid
AND second_row.a_time_start <= first_row.a_time_end
AND second_row.a_time_end >= first_row.a_time_start
),
eligible_before_seq AS (
SELECT independent_eligible.*
FROM independent_eligible
WHERE NOT EXISTS (
SELECT 1
FROM overlap_rows
WHERE overlap_rows.source_id = independent_eligible.source_id)
),
bad_groups AS (
SELECT a_date,
a_b_animid
FROM eligible_before_seq
GROUP BY a_date,
a_b_animid
HAVING MIN(a_seq_num) IS NULL
OR MIN(a_seq_num) <> 1
OR MAX(a_seq_num) IS DISTINCT FROM COUNT(*)
OR COUNT(DISTINCT a_seq_num) <> COUNT(*)
)
SELECT eligible_before_seq.*
FROM eligible_before_seq
JOIN bad_groups
USING (a_date, a_b_animid)
ORDER BY eligible_before_seq.a_date,
eligible_before_seq.a_b_animid,
eligible_before_seq.a_seq_num,
eligible_before_seq.a_time_start;
The refreshed projection contained 453 invalid animal/date groups and 910 rows after the other approved exclusions and the Problem #171 mapping.
Solution
Approved by the investigator on 2026-09-14: exclude every row in each invalid animal/date group. Do not renumber, retain a partial group, or select rows by order. Compute Problem #166 last, after Problems #165, #168 through #170, and
- 172 through #178,
and include the complete identification query beside the loader predicate.
Sanity must prove every remaining group has non-NULL sequence values exactly
equal to 1..N.
Implemented last in conversion/attendance_projection.sql by
attendance_bad_seq_groups and problem_166. The
dated 453 groups and 910 rows are audit evidence. Sanity proves every
refreshed invalid group is completely excluded and every eligible group is
exactly 1..N. Remove only when all post-exclusion groups are
contiguous.
(#167) ATTENDANCE observer values require PEOPLE support rows
Problem
The observer columns contain hundreds of case and edge-whitespace variants.
Observer codes must be matched by their standardized value so variants such as
Mike, mike, and Mike resolve to one
PEOPLE.Person. Standardized values absent from
PEOPLE require support rows.
Bad data
WITH raw_observers AS (
SELECT a_observer1_id::text AS raw_person
FROM clean.attendance
WHERE a_observer1_id IS NOT NULL
UNION
SELECT a_observer2_id::text
FROM clean.attendance
WHERE a_observer2_id IS NOT NULL
), standardized AS (
SELECT raw_person,
LOWER(NORMALIZE(BTRIM(raw_person))) AS standard_person
FROM raw_observers
)
SELECT standardized.standard_person,
ARRAY_AGG(standardized.raw_person
ORDER BY standardized.raw_person) AS raw_spellings,
ARRAY_AGG(people.person ORDER BY people.person)
FILTER (WHERE people.person IS NOT NULL) AS existing_codes
FROM standardized
LEFT JOIN codes.people
ON LOWER(NORMALIZE(BTRIM(people.person))) =
standardized.standard_person
GROUP BY standardized.standard_person
ORDER BY standardized.standard_person;
The refreshed 2026-09-14 snapshot had 932 distinct raw observer spellings in
718 standardized classes. Of those classes, 245 matched an existing
PEOPLE row and 473 required a new support row.
Solution
Approved by the investigator on 2026-09-14: standardize observer identifiers
with LOWER(NORMALIZE(BTRIM(value))). Resolve a class to its existing
PEOPLE.Person spelling when present. For an unmatched class, choose
one deterministic trimmed source spelling, preferring mixed case and then
bytewise order, and add it as an active support row. Use that spelling for
Person and Name and Attendance observer code: <code> for
Description. Map NULL to the existing NONE code. This support
decision excludes no rows except where Problem #168 also applies.
Implemented by attendance_observer_codes in
conversion/attendance_projection.sql and the ordered
PEOPLE insert in conversion/load_attendance.sql. Sanity
pins all 718 canonical codes using an explicitly byte-ordered, length-prefixed
serialization, all 932 raw-to-canonical mappings, and 473 missing support rows.
Remove only when every standardized observer value already resolves to one
active row.
* (#168) ATTENDANCE Recorder and Observer2 resolve to the same person code
Problem
ARRIVALS_A requires Recorder and Observer2 to differ. Some source
rows contain observer spellings that become equal after standardizing case and
edge whitespace; rows with both observers NULL also become equal after the
approved NULL-to-NONE mapping.
Bad data
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE LOWER(NORMALIZE(COALESCE(
BTRIM(attendance.a_observer1_id::text), 'NONE'))) =
LOWER(NORMALIZE(COALESCE(
BTRIM(attendance.a_observer2_id::text), 'NONE')))
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num,
attendance.a_time_start;
The 2026-09-14 snapshot returned 1,544 rows: 1,484 with equal trimmed non-NULL spellings, 39 additional case-equivalent pairs, and 21 with both observers NULL.
Solution
Approved by the investigator on 2026-09-14: exactly exclude every returned row. Do not fabricate or discard an observer. Include the query beside the Problem #168 predicate and prove no eligible detail has equal observer codes.
Implemented as problem_168 in
conversion/attendance_projection.sql. The dated 1,544 rows are
audit evidence; sanity proves that no refreshed eligible row has equal derived
observer codes. Remove only when no derived Recorder equals Observer2.
* (#169) ATTENDANCE focal does not resolve to BIOGRAPHY_DATA
Problem
An attendance watch and its Solo role require an exact production AnimID.
Thousands of source rows use values absent from BIOGRAPHY_DATA,
mostly NYANI. Case or whitespace repair is not approved.
Bad data
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE NOT EXISTS (
SELECT 1
FROM sokwedb.biography_data AS biography
WHERE biography.animid = attendance.a_b_animid::text)
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num,
attendance.a_time_start;
The exact-match query returned 6,877 rows on 2026-09-14.
Solution
Approved by the investigator on 2026-09-14: exactly exclude every returned row. Do not trim, case-fold, or map an unmatched focal to a sentinel. Include the query beside the Problem #169 predicate and prove every eligible focal resolves exactly.
Implemented as problem_169 in
conversion/attendance_projection.sql. The dated 6,877 rows are
audit evidence; sanity proves that every refreshed eligible focal resolves.
Remove only when every exact source focal resolves.
* (#170) ATTENDANCE focal is outside its study dates
Problem
A role participant must be under study on the event date. Some exact-matching focals have attendance dates before EntryDate or after DepartDate.
Bad data
SELECT attendance.*,
biography.entrydate,
biography.departdate
FROM clean.attendance AS attendance
JOIN sokwedb.biography_data AS biography
ON biography.animid = attendance.a_b_animid::text
WHERE attendance.a_date NOT BETWEEN
biography.entrydate AND biography.departdate
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
The 2026-09-14 snapshot returned six rows.
Solution
Approved by the investigator on 2026-09-14: exactly exclude every returned row. Include the query beside the Problem #170 predicate and prove every eligible participant is under study on the event date.
Implemented as problem_170 in
conversion/attendance_projection.sql. The dated six rows are audit
evidence; sanity proves that every refreshed eligible focal is under study on
its attendance date. Remove only when every exact focal is under study.
(#171) ATTENDANCE direction fields contain the -9 sentinel
Problem
Production degrees may be NULL or integers from 0 through 359. The source uses
-9 in one or both direction fields to mean that the direction was
not seen.
Bad data
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE COALESCE(attendance.a_degree_of_arrival = -9, FALSE)
OR COALESCE(attendance.a_degree_of_departure = -9, FALSE)
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
The 2026-09-14 snapshot returned 7,228 rows. Individual arrival and departure counts overlap and must not be summed.
Solution
Confirmed by the project investigators on 2026-09-15: translate each
-9 independently to NULL because it means the direction was not
seen. Preserve existing NULL and values from 0 through 359 exactly. Keep this
query beside the Problem #171 predicate to profile the mapped source condition,
but do not add Problem #171 to the exclusion array. Prove every eligible
target degree equals NULLIF(source_degree, -9) and satisfies the
target domain.
Implemented as the profiled problem_171 condition and the
NULLIF(..., -9) arrival/departure projection in
conversion/attendance_projection.sql. The dated 7,228 rows are
audit evidence; no rows are excluded solely by this issue. Sanity proves each
refreshed eligible target equals NULLIF(source_degree, -9). Remove
the mapping only when neither source degree contains -9.
* (#172) ATTENDANCE swelling values have no approved CYCLE_STATES mapping
Problem
Most source swelling spellings have an exact lossless representation in
CYCLE_STATES. The values -, 0.30, and
0.33 do not. Mapping - to MISS or rounding
the numeric values would change source meaning.
Bad data
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE BTRIM(attendance.a_type_of_cycle::text)
IN ('-', '0.30', '0.33')
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
The 2026-09-14 snapshot returned 3,238 rows: 3,221 with -, two with
0.30, and 15 with 0.33.
Solution
Approved by the investigator on 2026-09-14: exactly exclude every returned
row. Do not map or round these values. The explicit supported projection is:
0/0.00 to 0;
.25/0.25 to 0.25;
.5/0.5/0.50 to 0.5;
.75/0.75 to 0.75;
1/1.00 to 1;
n/a/N/A to n/a; and
u/U to U. Include the query beside the
Problem #172 predicate and reject any refreshed spelling outside this closed
mapping.
Implemented as problem_172 in
conversion/attendance_projection.sql. The dated 3,238 rows are
audit evidence; sanity rejects any refreshed eligible spelling outside the
closed exact mapping. Remove only when every source spelling belongs to it.
* (#173) ATTENDANCE CycleOld is missing or has edge spaces
Problem
ARRIVALS_A.CycleOld is required and may not contain only spaces.
The source has NULL values and edge-spaced values. Trimming would alter the
initial digitization recorded by this field.
Bad data
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE attendance.cycle_old IS NULL
OR attendance.cycle_old::text <>
BTRIM(attendance.cycle_old::text)
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
The 2026-09-14 snapshot returned 417 rows: 229 NULL and 188 edge-spaced values.
Solution
Approved by the investigator on 2026-09-14: exactly exclude every returned row. Preserve all eligible CycleOld text exactly; do not trim or invent a missing value. Include the query beside the Problem #173 predicate.
Implemented as problem_173 in
conversion/attendance_projection.sql. The dated 417 rows are audit
evidence; sanity proves every refreshed eligible CycleOld is non-NULL and
edge-space free. Remove only when every source value satisfies that rule.
* (#174) ATTENDANCE bananas contains a fractional value
Problem
ARRIVALS_A.Bananas is an integer. One source value is fractional,
and rounding or truncating it would alter the recorded value.
Bad data
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE attendance.a_bananas_given IS NOT NULL
AND attendance.a_bananas_given <>
TRUNC(attendance.a_bananas_given)
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
The 2026-09-14 snapshot returned one row with value 0.25.
Solution
Approved by the investigator on 2026-09-14: exactly exclude the returned row. Do not round or truncate it. Include the query beside the Problem #174 predicate and prove all eligible banana values are integral and in range.
Implemented as problem_174 in
conversion/attendance_projection.sql. The dated one-row result is
audit evidence; sanity proves all refreshed eligible banana values are
integral and in range. Remove only when all source values are integral.
* (#175) ATTENDANCE event interval is missing or inverted
Problem
Attendance event Start and Stop are required, and Start may not be after Stop. One source row has no Start and other rows have Start after Stop. Production attendance events cannot represent an inferred cross-midnight interval.
Bad data
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE attendance.a_time_start IS NULL
OR attendance.a_time_end IS NULL
OR attendance.a_time_start > attendance.a_time_end
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
The 2026-09-14 snapshot returned 28 rows: one missing Start, no missing Stop, and 27 inverted intervals.
Solution
Approved by the investigator on 2026-09-14: exactly exclude every returned row. Do not swap times, infer dates, derive Start from duration, or clamp a value. Include the query beside the Problem #175 predicate.
Implemented as problem_175 in
conversion/attendance_projection.sql. The dated 28 rows are audit
evidence; sanity proves every refreshed eligible interval has both endpoints
and is ordered. Remove only when all source intervals satisfy that rule.
* (#176) ATTENDANCE duration disagrees with the event interval
Problem
For otherwise ordered intervals, some stored durations differ from the exact minutes between Start and Stop. Production has no independent attendance duration destination, so loading only the times would discard a conflicting source assertion.
Bad data
SELECT attendance.*,
EXTRACT(EPOCH FROM
(attendance.a_time_end - attendance.a_time_start)) / 60
AS computed_minutes
FROM clean.attendance AS attendance
WHERE attendance.a_time_start IS NOT NULL
AND attendance.a_time_start <= attendance.a_time_end
AND attendance.a_duration_of_obs IS DISTINCT FROM
EXTRACT(EPOCH FROM
(attendance.a_time_end - attendance.a_time_start)) / 60
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
The 2026-09-14 snapshot returned 15 rows, separate from Problem #175.
Solution
Approved by the investigator on 2026-09-14: exactly exclude every returned row. Do not alter event times or discard the disagreement as warning-only. Include the query beside the Problem #176 predicate and prove exact agreement for every eligible row.
Implemented as problem_176 in
conversion/attendance_projection.sql. The dated 15 rows are audit
evidence; sanity proves every refreshed eligible duration agrees exactly with
elapsed time. Remove only when every otherwise valid source row agrees.
* (#177) ATTENDANCE intervals overlap for one animal and date
Problem
Production warns when one animal is recorded at the feeding station in overlapping intervals. Exact duplicates and independently invalid rows must be removed first so this issue identifies only otherwise eligible overlap participants.
Bad data
This query is intentionally complete. Keep its independent eligibility predicates synchronized with Problems #165 and #168 through #176/#178.
WITH source AS (
SELECT row_number() OVER () AS source_id,
attendance.*,
COUNT(*) OVER (
PARTITION BY attendance.a_date,
attendance.a_cl_community_id,
attendance.a_b_animid,
attendance.a_seq_num,
attendance.a_type_of_cycle,
attendance.a_observer1_id,
attendance.a_observer2_id,
attendance.a_bananas_given,
attendance.a_degree_of_arrival,
attendance.a_degree_of_departure,
attendance.a_time_start,
attendance.a_time_end,
attendance.a_duration_of_obs,
attendance.day,
attendance.mo,
attendance.yr,
attendance.cycle_old
) AS duplicate_count
FROM clean.attendance AS attendance
),
independent_eligible AS (
SELECT source.*
FROM source
JOIN sokwedb.biography_data AS biography
ON biography.animid = source.a_b_animid::text
WHERE source.duplicate_count = 1
AND source.a_date BETWEEN biography.entrydate AND biography.departdate
AND COALESCE(BTRIM(source.a_observer1_id::text), 'NONE')
<> COALESCE(BTRIM(source.a_observer2_id::text), 'NONE')
AND BTRIM(source.a_type_of_cycle::text)
NOT IN ('-', '0.30', '0.33')
AND source.cycle_old IS NOT NULL
AND source.cycle_old::text = BTRIM(source.cycle_old::text)
AND (source.a_bananas_given IS NULL
OR source.a_bananas_given = TRUNC(source.a_bananas_given))
AND source.a_time_start IS NOT NULL
AND source.a_time_end IS NOT NULL
AND source.a_time_start <= source.a_time_end
AND source.a_duration_of_obs IS NOT DISTINCT FROM
EXTRACT(EPOCH FROM
(source.a_time_end - source.a_time_start)) / 60
AND source.day IS NOT DISTINCT FROM
EXTRACT(DAY FROM source.a_date)::integer
AND source.mo IS NOT DISTINCT FROM
EXTRACT(MONTH FROM source.a_date)::integer
AND source.yr IS NOT DISTINCT FROM
EXTRACT(YEAR FROM source.a_date)::integer
)
SELECT DISTINCT first_row.*
FROM independent_eligible AS first_row
JOIN independent_eligible AS second_row
ON second_row.source_id <> first_row.source_id
AND second_row.a_date = first_row.a_date
AND second_row.a_b_animid = first_row.a_b_animid
AND second_row.a_time_start <= first_row.a_time_end
AND second_row.a_time_end >= first_row.a_time_start
ORDER BY first_row.a_date,
first_row.a_b_animid,
first_row.a_time_start,
first_row.a_time_end;
The refreshed projection returned 428 otherwise eligible source rows after Problem #171 became a mapping rather than an exclusion. The unfiltered source had 463 overlap pairs, a count that includes other issue classes and is not the exclusion baseline.
Solution
Approved by the investigator on 2026-09-14: exclude every row participating in an otherwise eligible overlap. Do not select one interval from a pair. The shared projection must define an execution-local source-row identity, include this complete query in code, and prove zero overlaps remain before Problem
- 166 sequence closure.
Implemented after independent exclusions as problem_177 in
conversion/attendance_projection.sql. The dated 428 rows are audit
evidence; sanity proves that no refreshed eligible intervals overlap. Remove
only when no otherwise eligible intervals overlap.
* (#178) ATTENDANCE date components disagree with A_date
Problem
The redundant source Day, Mo, and Yr values should describe
A_date. Some day or month values disagree. Discarding the
redundant values would hide a source conflict.
Bad data
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE attendance.day IS DISTINCT FROM
EXTRACT(DAY FROM attendance.a_date)::integer
OR attendance.mo IS DISTINCT FROM
EXTRACT(MONTH FROM attendance.a_date)::integer
OR attendance.yr IS DISTINCT FROM
EXTRACT(YEAR FROM attendance.a_date)::integer
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
The 2026-09-14 snapshot returned 79 distinct rows. Day disagreed in 46 rows, month in 33, and year in none.
Solution
Approved by the investigator on 2026-09-14: exactly exclude every returned
row. Do not silently prefer one component representation. Include the query
beside the Problem #178 predicate and prove all eligible redundant components
agree with A_date.
Implemented as problem_178 in
conversion/attendance_projection.sql. The dated 79 rows are audit
evidence; sanity proves every refreshed eligible redundant date component
agrees with A_date. Remove only when every source row agrees.
Problems #165, #166, #168 through #170, and #172 through #178 overlap and must not be summed. Problem #171 is profiled but mapped, not excluded. In the approved dependency order -- #165, independent #168-#170/#172-#176/#178,
- 177, then #166 -- the refreshed union excluded 13,714 rows and left 120,385
eligible rows. Refresh each issue, the overlap set, the final sequence closure, and the union before implementation.
On 2026-09-15 the investigator approved replacing complete-row fingerprints
and frozen issue-membership hashes with semantic validation. The documented
predicates and production contracts now determine acceptance. The conversion
uses an execution-local source_id to preserve row multiplicity and
prove relational parity; it is not a durable source identity. Historical
counts remain dated audit evidence and must be refreshed with
attendance_profile, including rebuilding the handoff Encounter Log
when testing a refreshed source.
(#179) ATTENDANCE observers require standardized PEOPLE matching
Problem
Exact matching would create separate PEOPLE rows for case and
edge-whitespace variants of the same observer identifier. This contradicts
the case-equivalent unique index and the established conversion convention.
Bad Data
Run against the refreshed clean, codes snapshot after
food support loading:
WITH raw_observers AS (
SELECT a_observer1_id::text AS raw_person
FROM clean.attendance
WHERE a_observer1_id IS NOT NULL
UNION
SELECT a_observer2_id::text
FROM clean.attendance
WHERE a_observer2_id IS NOT NULL
), standardized AS (
SELECT raw_person,
LOWER(NORMALIZE(BTRIM(raw_person))) AS standard_person
FROM raw_observers
)
SELECT standardized.standard_person,
ARRAY_AGG(standardized.raw_person
ORDER BY standardized.raw_person) AS raw_spellings,
ARRAY_AGG(people.person ORDER BY people.person)
FILTER (WHERE people.person IS NOT NULL) AS existing_codes
FROM standardized
LEFT JOIN codes.people
ON LOWER(NORMALIZE(BTRIM(people.person))) =
standardized.standard_person
GROUP BY standardized.standard_person
ORDER BY standardized.standard_person;
The complete refreshed profile contained 932 raw spellings in 718 standardized classes. Sixteen spellings had edge whitespace and 305 raw spellings mapped to a different canonical code after trimming and case-equivalent resolution.
Solution
Approved by the investigator on 2026-09-14: retain the case-equivalent
people_person_uniquenocase index and the edge-whitespace constraint.
Map every raw attendance spelling through the standardized key defined in
Problem #167. Existing PEOPLE.Person spellings take precedence;
each previously unseen standardized class receives one deterministic support
code. Sanity must prove every raw spelling maps exactly once and every
standardized class produces exactly one canonical code. This issue excludes
no rows except the newly recognized standardized observer equalities under
Problem #168.
(#180) load_finish deactivates ATTENDANCE observer codes
Problem
The existing finalization step deactivates NONE and every
PEOPLE.Person containing a slash. After attendance loading this
makes three referenced observer codes inactive: NONE,
N/A, and SELEMANI/YAHAYA. They account for 37,655
Recorder or Observer2 uses, violating the requirement that attendance observer
codes remain active.
Bad Data
Run after attendance and load_finish:
SELECT people.person,
COUNT(*) AS observer_uses
FROM (
SELECT arrivals_a.recorder AS person FROM sokwedb.arrivals_a
UNION ALL
SELECT arrivals_a.observer2 FROM sokwedb.arrivals_a
) AS attendance_observers
JOIN codes.people USING (person)
WHERE NOT people.active
GROUP BY people.person
ORDER BY people.person;
The refreshed query returned three codes and 37,655 observer uses:
N/A 54 times, NONE 37,599 times, and
SELEMANI/YAHAYA twice.
Solution
Approved by the investigator on 2026-09-14 as part of the active raw-observer
policy: conversion/load_finish.sql retains active status for any
person referenced by ARRIVALS_A.Recorder or
ARRIVALS_A.Observer2. Problem #39 finalization remains unchanged
for unreferenced UNK, NONE, and slash-containing codes.
This issue excludes no attendance rows. Remove the exception only when no
attendance detail references a code that finalization would deactivate.
(#181) COLOBUS descriptive text fields contain NULL
Problem
The Access COLOBUS source contains NULL in eight descriptive text
fields whose production destinations are required text columns. NULL occurs
in col_hunter_id 722 times, col_killer_id 1,438 times,
col_victim_age 1,447 times, col_colobus_details 1,652 times,
col_veg_details 2,073 times, col_hunt_failure_details 1,943 times,
col_comments 885 times, and col_hunt_description
1,803 times.
The seven colobus-specific values map to COLOBUS.Hunters,
COLOBUS.Killers, COLOBUS.VictimsAges, COLOBUS.Details,
COLOBUS.Vegetation, COLOBUS.FailureDetails, and COLOBUS.Description.
Comments map separately to the shared EVENTS.Notes column.
Bad Data
SELECT COUNT(*) FILTER (WHERE col_hunter_id IS NULL) AS hunters,
COUNT(*) FILTER (WHERE col_killer_id IS NULL) AS killers,
COUNT(*) FILTER (WHERE col_victim_age IS NULL) AS victims_ages,
COUNT(*) FILTER (WHERE col_colobus_details IS NULL) AS troop_details,
COUNT(*) FILTER (WHERE col_veg_details IS NULL) AS vegetation,
COUNT(*) FILTER (WHERE col_hunt_failure_details IS NULL)
AS failure_details,
COUNT(*) FILTER (WHERE col_comments IS NULL) AS comments,
COUNT(*) FILTER (WHERE col_hunt_description IS NULL) AS descriptions
FROM clean.colobus;
The refreshed query returned, in column order, 722, 1,438, 1,447, 1,652, 2,073, 1,943, 885, and 1,803 rows.
Solution
Approved by the investigator on 2026-09-15: retain the existing production
schema and map each NULL independently to the empty string. Implement each
destination as COALESCE(source_value, ); preserve every non-NULL
source value exactly. The existing required-text and not-only-spaces
constraints permit the empty string, so this normalization requires no schema
change. Do not concatenate comments with the encounter description, and do
not exclude a source row solely because any of these eight fields is NULL.
Sanity must prove exact field-by-field COALESCE(source_value, )
parity and zero exclusions attributable only to Problem #181. Remove this
normalization only when all eight source fields contain no NULL values.
(#182) MATING_EVENT community codes differ only by case
Problem
The Access MATING_EVENT.M_cl_community_id column contains the value kk,
which differs only by case from the existing production COMM_IDS.CommID code
KK. Production community codes are case-sensitive.
Bad Data
SELECT * FROM clean.mating_event WHERE m_cl_community_id = 'kk';
The refreshed source contains 181 such rows. All other mating community
values are KK or MT.
Solution
Approved by the investigators on 2026-09-15: standardize
m_cl_community_id = 'kk' to the existing KK code in the clean schema. This
is a case-only spelling normalization and excludes no rows. Preserve all
other source community values unchanged, and require the mating sanity check
to verify that every resulting community exists in COMM_IDS.
(#183) MATING_EVENT times may contain seconds
Problem
Production EVENTS.Start and EVENTS.Stop record times to minute precision
and reject nonzero seconds. The Access MATING_EVENT.M_time value is initially
represented as a timestamp, and merely converting it to a PostgreSQL time
value does not remove seconds.
Bad Data
SELECT * FROM raw."MATING_EVENT" WHERE extract(second FROM "M_time") <> 0;
The refreshed source contains one row, dated 2008-10-12 with focal GA, male
WL, female GA, and time 09:35:30. Problem #4 remains corrected: the refreshed
source has no non-midnight time component in M_FOL_date.
Solution
Approved by the investigators on 2026-09-15: mating event times are recorded
to minute precision. During construction of the tidy schema, truncate
M_time to the minute before tidy.sql converts the column to TIME. Do not
round the value. The mating sanity check must reject the conversion if a
value containing seconds nevertheless reaches the clean schema. This issue
excludes no rows and does not resolve missing or out-of-window mating times.
* (#184) MATING_EVENT ordinary boolean flags contain question marks
Problem
Production MATINGS.Incest, Consort, Guarding, Courting, and Camp are
required Boolean values. The Access fields use X for true and blank or NULL
for false, but one M_court_flag row and two different M_camp_flag rows
contain ?. The unknown values cannot be represented by required Booleans.
Bad Data
SELECT *
FROM clean.mating_event
WHERE BTRIM(m_court_flag) = '?'
OR BTRIM(m_camp_flag) = '?';
The refreshed source contains three distinct rows. No row contains ? in
both fields, and none overlaps the Problem #186 exclusion.
Solution
Approved by the investigators on 2026-09-15: map X to true and blank or NULL
to false for all five fields. Exclude the three rows containing ?. Sanity
must reject any other domain value. This issue independently excludes three
rows and contributes three rows to the exclusion union.
(#185) MATING_EVENT records two interference facts in one target field
Problem
M_interference records a description or chimp ID.
M_interference_too_late_flag records the ID of a chimp who interfered too
late. Production has only the single required text column
MATINGS.Interference, so copying or concatenating the fields without labels
would lose which fact each value represents.
Bad Data
SELECT COUNT(*) FILTER (
WHERE NULLIF(BTRIM(m_interference), '') IS NOT NULL
AND NULLIF(BTRIM(m_interference_too_late_flag), '') IS NOT NULL)
AS both_fields,
COUNT(*) FILTER (
WHERE NULLIF(BTRIM(m_interference), '') IS NOT NULL
AND NULLIF(BTRIM(m_interference_too_late_flag), '') IS NULL)
AS interference_only,
COUNT(*) FILTER (
WHERE NULLIF(BTRIM(m_interference), '') IS NULL
AND NULLIF(BTRIM(m_interference_too_late_flag), '') IS NOT NULL)
AS too_late_only
FROM clean.mating_event;
The refreshed source contains 82 rows with both fields, 3,760 with only
m_interference, and 135 with only the too-late field. No source
m_interference contains the labels Interference: or Too late:.
Solution
Approved by the investigator on 2026-09-16: trim both fields and use this
explicit encoding in MATINGS.Interference:
- neither field: the empty string;
m_interferenceonly: the trimmed source value;- too-late only:
Too late:followed by the trimmed source value; and - both fields:
Interference:followed by the trimmed interference value,
then ; Too late: followed by the trimmed too-late value.
This preserves the two source facts without changing the production schema.
Problem #186 separately excludes too-late X values before this encoding.
This issue excludes no rows.
* (#186) MATING_EVENT too-late interference contains X instead of an ID
Problem
The investigators identified M_interference_too_late_flag as the ID of a
chimp who interfered too late, but 110 source rows contain X. X is not a
chimp ID, and the source does not identify an individual who can replace it.
Bad Data
SELECT * FROM clean.mating_event WHERE BTRIM(m_interference_too_late_flag) = 'X';
The 110 rows span 1976 through 2015 and all use source B-REC. Eighty-two
also contain a separate interference description or ID; 28 do not. In
contrast, the 107 chimp-like values occur only in 2012 through 2015 and always
have an empty main interference field. The evidence is consistent with X
being an older marker, but it does not recover the required chimp ID.
Solution
Approved by the investigator on 2026-09-16: exclude all 110 rows containing
X in m_interference_too_late_flag. Do not interpret X as an individual
or silently discard the too-late fact. This issue independently excludes 110
rows, overlaps none of the three Problem #184 rows, and increases the
exclusion union from 3 to 113 rows.
* (#187) MATING_EVENT fail flags contain missing and unknown values
Problem
Production MATINGS.Fail is a required Boolean. The investigators confirmed
that X, Y, and y mean the mating was not complete and map to true, while
N maps to false. The Access source also contains blank, NULL, and ?
values, which require an explicit disposition.
Bad Data
SELECT COALESCE(NULLIF(BTRIM(m_fail_flag), ''), '[blank/NULL]') AS fail_value,
COUNT(*)
FROM clean.mating_event
GROUP BY COALESCE(NULLIF(BTRIM(m_fail_flag), ''), '[blank/NULL]')
ORDER BY fail_value;
The refreshed source contains 21,786 blank or NULL values and 552 ? values.
After Problems #184 and #186, 21,676 blank or NULL values and 551 ? values
remain eligible.
Solution
Approved by the investigator on 2026-09-16: map X, Y, and y to true;
map N, blank, and NULL to false; and exclude rows containing ?. Sanity
must reject any other domain value. The ? condition independently excludes
552 rows, overlaps one Problem #184 row and no Problem #186 rows, and increases
the exclusion union from 113 to 664 rows. It contributes 551 newly excluded
rows.
* (#188) MATING_EVENT times fall outside the permitted event window
Problem
Except for the separately governed sdb_no_time sentinel, production
EVENTS.Start cannot be before 04:00 and EVENTS.Stop cannot be after 20:00.
Some non-NULL Access mating times fall outside that interval.
Bad Data
SELECT *
FROM clean.mating_event
WHERE m_time IS NOT NULL
AND (m_time < '04:00'::TIME OR m_time > '20:00'::TIME)
ORDER BY m_fol_date, m_time;
The refreshed source contains 25 rows: 24 between 01:00 and 03:48 and one at 21:38. None overlaps Problems #184, #186, or #187.
Solution
Approved by the investigator on 2026-09-16: exclude the 25 rows with non-NULL times outside 04:00 through 20:00. Do not clamp, wrap, or reinterpret their times. This issue independently and newly excludes 25 rows, increasing the exclusion union from 664 to 689 rows.
(#189) MATING_EVENT rows may have no recorded time
Problem
The Access source contains 34 mating rows with NULL m_time. Production
EVENTS.Start and EVENTS.Stop are required. Before this decision, only
aggression, B-record-note, and pantgrunt events could use the established
sdb_no_time value to distinguish an unrecorded time from an observed time.
Bad Data
SELECT * FROM clean.mating_event WHERE m_time IS NULL;
The refreshed source contains 34 rows. One overlaps the prior exclusion union, so excluding missing times would discard 33 additional otherwise eligible mating records.
Solution
Approved by the investigator on 2026-09-16: permit mating events to use the
existing sdb_no_time sentinel. Add sdb_mating_event to the two owning
EVENTS constraint allowlists and document the expanded behavior set. At the
production load boundary, map only SQL NULL m_time values to sdb_no_time
for both EVENTS.Start and EVENTS.Stop; preserve every non-NULL source time.
The existing point-event and paired-sentinel constraints remain in force.
This issue excludes no rows.
(#190) MATING_EVENT swelling values may be missing
Problem
Production MATINGS.Swelling is required, but the Access source contains 572
NULL values and 16 ? values. The source context for the question marks
describes unavailable information, including "no estrous state noted" and
"swelling not recorded".
Bad Data
SELECT *
FROM clean.mating_event
WHERE m_fem_swelling IS NULL
OR BTRIM(m_fem_swelling) = '?';
After prior exclusions, 568 NULL values and all 16 question marks remain eligible.
Solution
Approved by the investigator on 2026-09-16: map both NULL and ? to the
existing CYCLE_STATES code MISS, documented as "Missing data". MISS has
NULL AsNum and SSRank, so these records do not contribute a measured value
to daily swelling minimum/maximum calculations. The original missing-value
encoding remains available in clean. This issue excludes no rows.
(#191) MATING_EVENT records one-third swelling
Problem
Nine otherwise eligible Access mating rows contain the swelling value 0.33,
which is absent from CYCLE_STATES. Inserting the value into MATINGS
reproduces a matings_swelling_fkey violation.
Bad Data
SELECT * FROM clean.mating_event WHERE BTRIM(m_fem_swelling) = '0.33';
Solution
Approved by the investigator on 2026-09-16: add CYCLE_STATES code 0.33
with AsNum 0.33, SSRank 3, and description "1/3 swollen". Shift the
existing 0.5, 0.75, 1, and U support rows to ranks 4 through 7 so the
unique integer rank continues to order measured swelling values correctly.
Map source 0.33 to the new exact code. This issue excludes no rows.
(#192) MATING_EVENT extractors require PEOPLE support
Problem
Production MATINGS.ExtractedBy is required and must reference an active
PEOPLE.Person. The source contains 20,747 NULL extractor values, one
case-only spelling difference, and three populated names absent from the
conversion PEOPLE table.
Bad Data
SELECT m_extracted_by, COUNT(*)
FROM clean.mating_event
WHERE NULLIF(BTRIM(m_extracted_by), '') IS NULL
OR NOT EXISTS (
SELECT 1
FROM clean.people
WHERE LOWER(NORMALIZE(BTRIM(people.person))) =
LOWER(NORMALIZE(BTRIM(mating_event.m_extracted_by))))
GROUP BY m_extracted_by
ORDER BY m_extracted_by;
Karen McLellan matches the existing stored spelling KAREN MCLELLAN.
Deus Mjungu, FELDBLUM, and Joclyn Antonio have no standardized match.
Solution
Approved by the investigator on 2026-09-16: map a NULL, empty, or
whitespace-only extractor to UNK; trim populated values; reuse the exact
stored PEOPLE.Person spelling for standardized case-equivalent matches; and
add Deus Mjungu, FELDBLUM, and Joclyn Antonio as active support people.
Use LOWER(NORMALIZE(BTRIM(value))) as the deterministic matching key. This
issue excludes no rows.
(#193) MATING_EVENT comments require a production destination
Problem
The Access source stores mating comments separately from the full mating
description. Production has one shared event-notes field, EVENTS.Notes,
and silently concatenating the two source fields would erase that distinction.
The source contains 14,294 NULL comments and 15,134 comments that are NULL or
empty after trimming.
Bad Data
SELECT COUNT(*) FILTER (WHERE m_comments IS NULL) AS null_comments,
COUNT(*) FILTER (
WHERE NULLIF(BTRIM(m_comments), '') IS NULL) AS blank_comments
FROM clean.mating_event;
Solution
Approved by the investigator on 2026-09-16: map m_comments alone to
EVENTS.Notes. Map SQL NULL to the empty string with
COALESCE(m_comments, ) and preserve every non-NULL source value exactly.
Do not concatenate m_full_description into event notes. This issue excludes
no rows.
(#194) MATING_EVENT full descriptions require a mating-specific destination
Problem
The Access source stores a full mating description in m_full_description.
BRECORD_NOTES cannot preserve it because that detail table can attach only
to a B-record event, cannot share the mating EID, and requires unrelated
B-record text fields. The source contains 18,212 descriptions that are NULL
or empty after trimming.
Bad Data
SELECT COUNT(*) FILTER (WHERE m_full_description IS NULL)
AS null_descriptions,
COUNT(*) FILTER (
WHERE NULLIF(BTRIM(m_full_description), '') IS NULL)
AS blank_descriptions
FROM clean.mating_event;
Solution
Approved by the investigator on 2026-09-16: add the required text column
MATINGS.Description and map m_full_description to it. Map SQL NULL to the
empty string with COALESCE(m_full_description, ) and preserve every
non-NULL source value exactly. The production constraint permits the empty
string but rejects nonempty whitespace-only values.
The permanent schema and documentation change was implemented as append-only
master commit a628fdd (feat(matings): preserve mating descriptions) and
fast-forwarded to local, origin, private, and blessed master. This issue
excludes no rows.
(#195) MATING_EVENT sources require MATING_SOURCES support
Problem
Production MATINGS.Source is required and references
MATING_SOURCES.Source, but the source code table is empty. The Access source
contains mixed-case source labels while the production support key requires
uppercase values.
Bad Data
SELECT m_source, COUNT(*) AS row_count,
MIN(m_fol_date) AS first_date,
MAX(m_fol_date) AS last_date
FROM clean.mating_event
GROUP BY m_source
ORDER BY m_source;
The refreshed source contains B-REC 26,066 times over 1976 through 2018,
Tiki 393 times over 2000 through 2007, and Bobu 24 times in 2000. The
repository provides no more authoritative definition of Bobu than its
source label.
Solution
Approved by the investigator on 2026-09-16: add literal support codes B-REC,
TIKI, and BOBU, described respectively as "Mating record sourced from
B-REC", "Mating record sourced from Tiki", and "Mating record sourced from
Bobu". At the production boundary map source B-REC to B-REC, Tiki to
TIKI, and Bobu to BOBU. Do not claim semantics beyond the source labels.
Sanity must reject any refreshed source value outside this approved domain and
verify that every transformed code exists exactly once. This issue excludes
no rows.
* (#196) MATING_EVENT participants are absent from BIOGRAPHY_DATA
Problem
Production ROLES.Participant is required and references
BIOGRAPHY_DATA.AnimID. The first exclusion-free loader run failed while
inserting female participant STR for conversion identity 117. The source
also contains other participant values that are not production biography
identifiers, including category labels, unknown markers, blanks, and
unresolved spellings.
Bad Data
WITH eligible AS (
SELECT *
FROM clean.mating_event
WHERE COALESCE(BTRIM(m_court_flag), '') <> '?'
AND COALESCE(BTRIM(m_camp_flag), '') <> '?'
AND COALESCE(BTRIM(m_interference_too_late_flag), '') <> 'X'
AND COALESCE(BTRIM(m_fail_flag), '') <> '?'
AND (m_time IS NULL
OR m_time BETWEEN '04:00'::TIME AND '20:00'::TIME)
)
SELECT DISTINCT eligible.m_conversion_id
FROM eligible
CROSS JOIN LATERAL (
VALUES (eligible.m_male), (eligible.m_female)
) AS participant(animid)
WHERE NOT EXISTS (
SELECT 1
FROM sokwedb.biography_data
WHERE biography_data.animid = BTRIM(participant.animid))
ORDER BY eligible.m_conversion_id;
The refreshed post-follow database contains 574 missing participant
occurrences in 566 otherwise eligible source rows, representing 34 distinct
trimmed values. The first failure is identity 117, dated 2000-10-14, with
focal BE, male SL, and female STR. The failed first chunk rolled back
and left zero MATE events.
Solution
Approved by the investigator on 2026-09-16: exclude every source row for which
either trimmed participant is absent from BIOGRAPHY_DATA. Do not create
biography rows or substitute participant identities without source evidence.
Sanity must verify that this condition selects exactly 566 distinct eligible
source identities in the retained snapshot. The rows overlap none of the
prior 689-row exclusion union and increase it to 1,255 rows, leaving 25,228
rows eligible before later failures.
* (#197) MATING_EVENT participants fall outside biography study dates
Problem
After excluding Problem #196 rows, the loader failed when actor FAM
participated on 2014-09-10, after the individual's 2012-04-26 departure date.
Production requires every role participant to be biography-valid on the event
date.
Bad Data
SELECT DISTINCT mating_event.m_conversion_id
FROM clean.mating_event
CROSS JOIN LATERAL (
VALUES (mating_event.m_male), (mating_event.m_female)
) AS participant(animid)
JOIN sokwedb.biography_data
ON biography_data.animid = BTRIM(participant.animid)
WHERE mating_event.m_fol_date < biography_data.entrydate
OR mating_event.m_fol_date > biography_data.departdate
ORDER BY mating_event.m_conversion_id;
After applying Problems #184, #186, #187, #188, and #196, 28 participant
occurrences in 28 source rows fall outside study dates. Three are male actors
after departure, 13 are female actees after departure, and 12 are female
actees before entry. The affected participants are AT, CF, FAM, OR,
SF, SH, SIF, SP, WN, and YD. The failed first chunk rolled back
and left zero MATE events.
Solution
Approved by the investigator on 2026-09-16: exclude all 28 rows. Do not alter source dates or biography study spans. Sanity must verify the exact 28-row condition after the prior exclusions. This increases the exclusion union from 1,255 to 1,283 rows and leaves 25,200 rows eligible before later failures.
* (#198) MATING_EVENT focal values are absent from BIOGRAPHY_DATA
Problem
After Problems #196 and #197, the loader failed while creating an Other watch
for focal LUT, which is absent from BIOGRAPHY_DATA. Production requires
every WATCHES.AnimID, including mating-only Other watches, to be a biography
identifier or the established no-focal sentinel.
Bad Data
SELECT COALESCE(NULLIF(BTRIM(m_fol_b_focal_animid), ''), 'NONE') AS focal,
COUNT(*) AS row_count
FROM clean.mating_event
WHERE NOT EXISTS (
SELECT 1
FROM sokwedb.biography_data
WHERE biography_data.animid = COALESCE(
NULLIF(BTRIM(mating_event.m_fol_b_focal_animid), ''), 'NONE'))
GROUP BY COALESCE(NULLIF(BTRIM(m_fol_b_focal_animid), ''), 'NONE')
ORDER BY focal;
After the prior exclusions, 311 otherwise eligible rows use 51 unsupported
focal values. Three source rows have unique case-insensitive biography
matches: Gb to GB, lam to LAM, and sif to SIF. The remaining 308
rows use 48 unsupported values, many representing groups, multiple animals,
or missing-target markers. The failed first chunk rolled back and left zero
MATE events.
Solution
Approved by the investigator on 2026-09-16: normalize only the three proven
case variants in clean, then exclude the 308 rows whose normalized focal is
still absent from BIOGRAPHY_DATA. Do not infer individual focal identities
from group or multi-animal values. Sanity must verify the exact 308-row
residual after prior exclusions. This increases the exclusion union from
1,283 to 1,591 rows and leaves 24,892 rows eligible before later failures.
* (#199) MATING_EVENT focal dates are outside BIOGRAPHY_DATA study dates
Problem
After Problem #198, the loader failed while creating an Other watch for focal
POR on 2016-09-22. POR departed on 2014-03-13, and production requires a
watch date to fall within the focal animal's biography study dates.
Bad Data
SELECT mating_event.m_conversion_id,
mating_event.m_fol_date,
focal_biography.animid,
focal_biography.entrydate,
focal_biography.departdate
FROM clean.mating_event
JOIN sokwedb.biography_data AS focal_biography
ON focal_biography.animid = COALESCE(
NULLIF(BTRIM(mating_event.m_fol_b_focal_animid), ''), 'NONE')
WHERE mating_event.m_fol_date < focal_biography.entrydate
OR mating_event.m_fol_date > focal_biography.departdate
ORDER BY mating_event.m_conversion_id;
After the prior exclusions, 43 otherwise eligible rows have focal dates
outside the focal's study dates. Forty-one rows use POR after departure;
two use SN before entry. The failure occurred after 16,000 rows had
committed in earlier chunks, while the failing chunk rolled back.
Solution
Approved by the investigator on 2026-09-16: exclude all 43 rows. Do not alter source dates or biography study spans. Sanity must verify the exact 43-row condition after the prior exclusions. This increases the exclusion union from 1,591 to 1,634 rows and leaves 24,849 rows eligible before later failures. Rebuild the post-follow baseline before retrying so committed MATE rows and any mating-created Other watches are removed together.
(#200) COLOBUS encounters have no recorded Stop
Problem
There are 204 COLOBUS source rows with a recorded Start but no Stop. The
general EVENTS sentinel rules did not permit preserving a known Start with an
unknown Stop for behavior COL.
Bad Data
SELECT col_encounter_date, col_fol_b_focal_chimp_id, col_begin_time FROM clean.colobus WHERE col_end_time IS NULL ORDER BY col_encounter_date, col_fol_b_focal_chimp_id, col_begin_time;
Solution
Approved for this conversion: retain each recorded Start and map only the
missing Stop to sdb_no_time. Add a narrow, documented EVENTS exception for
COL with a non-sentinel Start and sentinel Stop. Do not exclude these rows or
allow a sentinel COL Start. Validation must find exactly 204 converted COL
events with the sentinel Stop.
* (#201) COLOBUS encounter times violate the approved time range
Problem
Fifteen rows cannot be represented without changing a recorded event time: eight have Start after Stop, two have Start after 20:00, and seven have Stop after 20:00. The two late Starts overlap the Start-after-Stop class, so the distinct union is 15 rows. None has a missing Stop.
Bad Data
SELECT col_encounter_date, col_fol_b_focal_chimp_id,
col_begin_time, col_end_time
FROM clean.colobus
WHERE col_begin_time > col_end_time
OR col_begin_time > '20:00'::TIME
OR col_end_time > '20:00'::TIME
ORDER BY col_encounter_date, col_fol_b_focal_chimp_id, col_begin_time;
Solution
Approved for this conversion: exclude exactly this 15-row union at the loader boundary and preserve all other recorded times. Sanity must fail if the predicate no longer identifies exactly 15 rows. This establishes an exclusion union of 15 rows before Problem #210.
(#202) COLOBUS StartMap is missing from seven rows
Problem
Seven source rows have no COL_begin_time_MAP. Of the 2,240 recorded values,
126 intentionally differ from the standard 15-minute calculation and must not
be recalculated.
Bad Data
SELECT COUNT(*) FILTER (WHERE col_begin_time_map IS NULL) AS missing,
COUNT(*) FILTER (
WHERE col_begin_time_map IS NOT NULL
AND col_begin_time_map IS DISTINCT FROM
DATE_BIN('15 minutes',
DATE '1960-07-04' + col_begin_time
+ INTERVAL '8 minutes',
TIMESTAMP '1960-07-04 00:00')::TIME) AS adjusted
FROM clean.colobus;
Solution
Approved for this conversion: derive only the seven missing values at the
production load boundary with the displayed DATE_BIN expression. Preserve
all recorded values exactly. This excludes no rows.
(#203) COLOBUS Hunt, HuntCert, and Kill can be unknown
Problem
The source contains 15 NULL Hunt flags, 16 NULL HuntCert flags, and 221 NULL
Kill flags, but the corresponding production columns were NOT NULL.
Bad Data
SELECT COUNT(*) FILTER (WHERE col_hunt_flag IS NULL) AS hunt_unknown,
COUNT(*) FILTER (WHERE col_hunt_cert IS NULL) AS huntcert_unknown,
COUNT(*) FILTER (WHERE col_kill_flag IS NULL) AS kill_unknown
FROM clean.colobus;
Solution
Approved for this conversion: make all three production columns nullable,
map Y to true and N to false, and preserve source NULL as SQL NULL. This
excludes no rows.
(#204) COLOBUS FocalHuntCert can be unknown
Problem
There are 744 NULL focal-hunt-certainty values. Documentation described an
unknown state, but the production column was NOT NULL.
Bad Data
SELECT COUNT(*) FROM clean.colobus WHERE col_focal_hunt_cert IS NULL;
Solution
Approved for this conversion: resolve the schema/documentation conflict in
favor of the documented unknown state. Make FocalHuntCert nullable, map
Y and N to booleans, and preserve all source NULLs. This excludes no
rows.
(#205) COLOBUS NumKills is absent for several distinct states
Problem
There are 389 rows without a total kill count: seven have Kill = 'Y', 163
have Kill = 'N', and 219 also have an unknown Kill flag. Seven meaningful
recorded male totals exceed the recorded overall total and must not be
rewritten.
Bad Data
SELECT col_kill_flag, COUNT(*) FROM clean.colobus WHERE col_num_kills_total IS NULL GROUP BY col_kill_flag ORDER BY col_kill_flag NULLS LAST;
Solution
Approved for this conversion: make NumKills nullable. Preserve every
recorded total; for a missing total map Kill = 'Y' to 1, Kill = 'N' to 0,
and unknown Kill to SQL NULL. Preserve the seven male/total differences and
verify source-to-target parity. This excludes no rows.
(#206) COLOBUS FemaleNumKills has no supported derivation
Problem
The source does not provide an independently supported total female kill count. Deriving it from other totals would invent data and can be invalid for the seven rows where male kills exceed the overall total.
Bad Data
SELECT COUNT(*)
FROM clean.colobus
WHERE col_num_kills_total IS NOT NULL
AND col_num_kills_by_males > col_num_kills_total;
The query returns seven rows.
Solution
Approved for this conversion: make FemaleNumKills nullable and always load
SQL NULL. Do not derive it by subtraction. This excludes no rows.
(#207) COLOBUS focal kill counts contain fractions
Problem
Three focal-male and three focal-female source counts have fractional values.
The production INTEGER columns cannot preserve them.
Bad Data
SELECT COUNT(*) FILTER (
WHERE col_num_kills_by_focal_male <>
TRUNC(col_num_kills_by_focal_male)) AS focal_male,
COUNT(*) FILTER (
WHERE col_num_kills_by_focal_female <>
TRUNC(col_num_kills_by_focal_female)) AS focal_female
FROM clean.colobus;
Solution
Approved for this conversion: change FocalMaleNumKills and
FocalFemaleNumKills to exact NUMERIC columns and preserve each source
value. This excludes no rows.
(#208) COLOBUS group-size codes are unsupported and can be absent
Problem
The support table was empty. Source values use codes 1 through 5, and 611
rows have no group-size code. Missing values must not be changed to code 4,
which has a distinct recorded meaning.
Bad Data
SELECT col_colobus_group_size, COUNT(*) FROM clean.colobus GROUP BY col_colobus_group_size ORDER BY col_colobus_group_size NULLS LAST;
The counts are 18, 3, 11, 1,555, and 49 for codes 1 through 5, plus 611
NULLs.
Solution
Approved for this conversion: load the five documented support rows, retain
the foreign key, make GroupSize nullable, and preserve source NULL as SQL
NULL. The descriptions are group consisted of only 1 col, group consisted
of only 2 col, group defined as small by observer, group but no size
estimate given, and group defined as large. This excludes no rows.
(#209) COLOBUS focal identifiers require approved corrections
Problem
Five source rows use four noncanonical focal identifiers. The approved
corrections are GRP to NONE, STR to MGF, U to UNK, and FLT to
FLI. NONE does not fit the source column's VARCHAR(3) declaration.
Bad Data
SELECT col_fol_b_focal_chimp_id, COUNT(*)
FROM clean.colobus
WHERE col_fol_b_focal_chimp_id IN ('GRP', 'STR', 'U', 'FLT')
GROUP BY col_fol_b_focal_chimp_id
ORDER BY col_fol_b_focal_chimp_id;
Solution
Approved for this conversion: widen only the clean copy of the focal column to
TEXT, then apply the four corrections to the five affected rows. Preserve
raw, tidy, and easy source fidelity. This correction itself excludes no
rows; the two corrected UNK rows are handled independently by Problem #210.
* (#210) Corrected COLOBUS unknown focals have no valid watch context
Problem
After Problem #209 and all preceding watch loaders, the corrected UNK rows
at 1982-10-30 09:10 and 1985-07-21 10:12 have neither a B/Other watch nor a
covering COMM_MEMBS row. Inferring a community from unrelated same-day
observations would create unsupported data.
Bad Data
SELECT col_encounter_date, col_fol_b_focal_chimp_id, col_begin_time
FROM clean.colobus
WHERE col_fol_b_focal_chimp_id = 'UNK'
AND (col_encounter_date, col_begin_time) IN (
(DATE '1982-10-30', TIME '09:10'),
(DATE '1985-07-21', TIME '10:12'))
ORDER BY col_encounter_date, col_begin_time;
Solution
The investigator approved excluding exactly these two rows. Do not create a watch with an inferred community. They do not overlap Problem #201, raising the exact exclusion union from 15 to 17 rows. Of 2,247 source rows, 2,230 are therefore eligible for conversion.
* (#211) Corrected COLOBUS no-focal row has conflicting community context
Problem
After Problem #209 and all preceding watch loaders, the corrected GRP to
NONE row at 1978-09-10 11:33 has neither a B/Other watch nor a covering
COMM_MEMBS row. Its source comment says ON PATROL, KAHAMA. NO FOCAL, but
the unique same-day attendance community is KK (Kasekela); HK is Kahama
and its support notes say that community ended on 1977-12-31. Either community
choice would therefore infer unsupported data.
Bad Data
SELECT col_encounter_date, col_fol_b_focal_chimp_id,
col_begin_time, col_comments
FROM clean.colobus
WHERE col_fol_b_focal_chimp_id = 'NONE'
AND col_encounter_date = DATE '1978-09-10'
AND col_begin_time = TIME '11:33';
Solution
The investigator approved excluding exactly this row. Do not create a no-focal Other watch with an inferred community. It does not overlap Problems
- 201 or #210, raising the exact exclusion union from 17 to 18 rows. Of 2,247
source rows, 2,229 are therefore eligible for conversion.
(#212) OTHER_SPECIES requires production species support codes
Problem
codes.species is empty, while clean.other_species_lookup contains 15
unique local-species names and unique English descriptions. Every
SPECIES_PRESENT.Species value must reference a nonempty uppercase
codes.species.Species value.
Bad Data
SELECT UPPER(BTRIM(osl_local_species_name)) AS proposed_species,
osl_english_species_name AS proposed_description
FROM clean.other_species_lookup
ORDER BY proposed_species;
The query returns 15 rows: CHATU, FUNGO, KAKAKUONA, KENGE, KIMA,
MBOGO, NGURUWE, NKUNGE, NYANI, NYOKA, PONGO, TUMBILI,
UNKNOWN, UNRECORDED, and VYONDI.
Solution
Approved by the investigator on 2026-09-18: populate codes.species using
the uppercase, trimmed local lookup name as Species and the corresponding
English lookup name as Description. Preserve all 15 descriptions exactly,
including Mongoose? Civet? TBD. Sanity must verify the exact 15 pairs and
their uniqueness before loading. This excludes no rows.
(#213) OTHER_SPECIES species labels require canonical standardization
Problem
The source stores 30 distinct spellings of its classified local-species field.
Trimmed, case-insensitive matching resolves 6,360 of 6,366 rows to the 15-row
lookup. One additional row uses Viondi, a spelling variant of lookup value
Vyondi. The production code must use the uppercase canonical lookup value.
Bad Data
SELECT quote_literal(os_osl_local_species_name) AS source_value,
COUNT(*) AS rows
FROM clean.other_species AS other_species
WHERE NOT EXISTS (
SELECT 1
FROM clean.other_species_lookup AS lookup
WHERE UPPER(BTRIM(lookup.osl_local_species_name)) =
UPPER(BTRIM(other_species.os_osl_local_species_name)))
GROUP BY os_osl_local_species_name
ORDER BY source_value;
The query returns Sokwe mgeni (4), UMBA (1), and Viondi (1).
Solution
Approved by the investigator on 2026-09-18: standardize classified source
labels by trimmed, case-insensitive lookup and use the lookup's uppercase
local name as the production code. Correct the single Viondi value to
canonical VYONDI. Do not use OS_local_species_name_written to override
the classified field. This resolves 6,361 rows and excludes none. The five
remaining unlisted rows are handled independently by Problem #214.
* (#214) OTHER_SPECIES contains unlisted species labels
Problem
Four source rows are classified as Sokwe mgeni, and one is classified as
UMBA with comment 'umba' written on sheet. Neither trimmed label exists
in clean.other_species_lookup or has an approved codes.species code or
description.
Bad Data
SELECT os_fol_date, os_fol_b_focal_animid, os_time_begin,
os_time_end, os_osl_local_species_name,
os_local_species_name_written, os_comments
FROM clean.other_species AS other_species
WHERE UPPER(BTRIM(os_osl_local_species_name)) IN ('SOKWE MGENI', 'UMBA')
ORDER BY os_fol_date, os_fol_b_focal_animid, os_time_begin;
The five rows overlap neither the Problem #215 focal exclusion nor the Problem
- 216 time exclusion.
Solution
Approved by the investigator on 2026-09-18: exclude exactly these five rows at
the loader boundary. Do not map either label to UNKNOWN or UNRECORDED and
do not create unsupported species codes. These rows overlap none of Problems
- 215, #216, or #219 and contribute five rows to the final 13-row exclusion
union.
* (#215) OTHER_SPECIES focal identifiers require normalization and one exclusion
Problem
Twelve rows have trailing spaces in OS_FOL_B_focal_AnimID, one row uses
lowercase pl, and one row uses IMB. The trimmed values resolve uniquely
to canonical production IDs for the 12 edge-spaced rows, and pl resolves
uniquely to PL. IMB is absent from BIOGRAPHY_DATA and has no watch.
Although an IMA B watch exists on the same date, that does not prove the two
identifiers name the same individual.
Bad Data
SELECT DISTINCT quote_literal(os_fol_b_focal_animid) AS source_focal,
array_agg(DISTINCT biography_data.animid
ORDER BY biography_data.animid)
FILTER (WHERE biography_data.animid IS NOT NULL) AS canonical_matches
FROM clean.other_species
LEFT JOIN sokwedb.biography_data
ON LOWER(biography_data.animid) =
LOWER(BTRIM(other_species.os_fol_b_focal_animid))
WHERE os_fol_b_focal_animid <> BTRIM(os_fol_b_focal_animid)
OR NOT EXISTS (
SELECT 1
FROM sokwedb.biography_data AS exact_biography
WHERE exact_biography.animid =
other_species.os_fol_b_focal_animid)
GROUP BY os_fol_b_focal_animid
ORDER BY source_focal;
Solution
Approved by the investigator on 2026-09-18: resolve trimmed values to the
single case-insensitive canonical BIOGRAPHY_DATA.AnimID, preserving its
stored spelling. This removes the 12 trailing spaces and maps pl to PL.
Exclude exactly the IMB row dated 2016-08-27 at 10:30; do not infer IMA.
Normalized source keys remain 6,366 unique. Exclude exactly the IMB row;
the independently missing B watch for FU is handled by Problem #219.
* (#216) OTHER_SPECIES event times cannot satisfy production constraints
Problem
Six source rows cannot be represented without inventing or rewriting event times. Three have NULL end times, two have Start after Stop, and one additional row starts before 04:00. One reversed row also starts after 20:00. There are no second-bearing times and no Stop values after 20:00.
Bad Data
SELECT os_fol_date, os_fol_b_focal_animid,
os_time_begin, os_time_end, os_duration,
os_osl_local_species_name
FROM clean.other_species
WHERE os_time_end IS NULL
OR os_time_begin > os_time_end
OR os_time_begin < TIME '04:00'
OR os_time_begin > TIME '20:00'
OR os_time_end > TIME '20:00'
ORDER BY os_fol_date, os_fol_b_focal_animid, os_time_begin;
Solution
Approved by the investigator on 2026-09-18: exclude the exact six-row union
at the loader boundary. Do not derive a missing endpoint from Duration or use
Duration to replace a recorded time. Preserve both endpoints for every other
row; OS events are intervals and cannot use sdb_no_time. These six rows
do not overlap Problems #214, #215, or #219. Together with Problems #214 and
- 215 they bring the exclusion union to 12 rows.
(#217) OTHER_SPECIES comments map to event notes
Problem
EVENTS.Notes is NOT NULL. Of 6,366 source rows, 6,352 have NULL
OS_comments and 14 have non-NULL comments. The separate
OS_local_species_name_written field contains free-form audit evidence but
has no production destination and sometimes differs from the classified
local-species field.
Bad Data
SELECT COUNT(*) FILTER (WHERE os_comments IS NULL) AS null_comments,
COUNT(*) FILTER (WHERE os_comments IS NOT NULL) AS recorded_comments,
COUNT(*) FILTER (
WHERE os_comments IS NOT NULL
AND os_comments <> ''
AND BTRIM(os_comments) = '') AS whitespace_only_comments
FROM clean.other_species;
The query returns 6,352 NULL comments, 14 recorded comments, and zero nonempty whitespace-only comments.
Solution
Approved by the investigator on 2026-09-18: map OS_comments to
EVENTS.Notes, preserving non-NULL text exactly and mapping SQL NULL to .
Retain OS_local_species_name_written in clean as audit evidence; do not
append it to event notes or use it to override the classified species field.
This excludes no rows. Validation must prove two-way parity under
COALESCE(os_comments, ).
(#218) SPECIES_PRESENT trigger error paths reference nonexistent fields
Problem
Both error-detail paths in
db/schemas/lib/triggers/create/species_present.m4 reference
NEW.researchers and NEW.nonresearchers. Those are HUMANS columns;
SPECIES_PRESENT has only EID and Species. An invalid related event or a
conflicting HUMANS row can therefore obscure the intended integrity error
with a record-field error.
Bad Data
grep -nE 'NEW\.(researchers|nonresearchers)' \ db/schemas/lib/triggers/create/species_present.m4
The command reports four references, two in each error-detail path.
Solution
Resolved on 2026-09-18. Both invalid attempted-row details now report
NEW.species; the related event/watch and conflicting HUMANS details remain.
The generated trigger was installed in a disposable PostgreSQL 18.6 database,
and db/tests/species_present_trigger.sql passed rollback-only tests for wrong
event behavior, HUMANS conflict, a valid OS insert, immediate deferred
constraints, and fixture cleanup. This is a trigger repair and excludes no
source rows.
* (#219) OTHER_SPECIES row has no B watch at the required load stage
Problem
After the normal conversion sequence through load_follow_to_watches, the
otherwise valid FU row dated 2015-12-13 at 16:45 has no B watch. FU has no
source FOLLOW row that day. Later aggression and grooming loaders create a B
watch for the same focal/date, but moving OTHER_SPECIES later would make its
prerequisite depend on unrelated observations and conceal the stage-specific
source gap.
Bad Data
SELECT os_fol_date, os_fol_b_focal_animid, os_time_begin, os_time_end,
os_osl_local_species_name, os_comments
FROM clean.other_species
WHERE os_fol_date = DATE '2015-12-13'
AND os_fol_b_focal_animid = 'FU'
AND os_time_begin = TIME '16:45';
SELECT watches.wid
FROM sokwedb.watches
WHERE watches.animid = 'FU'
AND watches.date = DATE '2015-12-13'
AND watches.type = 'B';
At the required post-follow loader stage, the first query returns the one
VYONDI interval from 16:45 through 17:00 and the second query returns no row.
Solution
Approved by the investigator on 2026-09-18: exclude exactly the FU row dated
2015-12-13 at 16:45. Keep OTHER_SPECIES immediately after
load_follow_to_watches; do not create a watch and do not reuse a watch that
only appears after later aggression or grooming loads. This row overlaps
none of Problems #214, #215, or #216, increasing the final exclusion union to
13 rows and leaving 6,353 rows eligible for conversion.
(#220) FOLLOW_MAP_LOCATION rows can contain two location representations
Problem
Of 1,823,766 FOLLOW_MAP_LOCATION rows, 703,807 contain both a paper map
sequence and paired UTM coordinates. Production stores PAPER and UTM details
on separate EVENTS rows, so treating the source row as a choice of one detail
table would discard one recorded representation.
Bad Data
The read-only conversion/follow_map_location_profile.sql reports 703,807
dual rows, 69 paper-only rows, 1,119,890 UTM-only rows, and no row with neither
representation. Before exclusions this is 703,876 PAPER representations and
1,823,697 UTM representations.
Solution
Approved by the investigator on 2026-09-18: emit one PAPER event and one UTM event for every eligible dual row. Both events use the same location watch and source time. No loader may choose one representation or infer that UTM coordinates accompanying a paper map are direct GPS observations.
* (#221) FOLLOW_MAP_LOCATION focal identities are missing or unsupported
Problem
Location watches require a real BIOGRAPHY_DATA.AnimID; NONE is prohibited.
The refreshed source has 6,509 NULL-focal rows and 7,495 rows in 33 trimmed
labels having no normalized biography match. The source does not contain
evidence from which to invent their identities.
Fourteen noncanonical spellings covering 609 rows do resolve uniquely under
lower(normalize(btrim(value))). They include edge-space variants and the
case variants Nas to NAS, fd to FD, pl to PL, and sw to SW.
Bad Data
Run conversion/follow_map_location_profile.sql. It prints every unmatched
trimmed label with its row count and date range, and reports 214 raw non-NULL
spellings, 204 trimmed spellings, and 200 normalized spelling classes. The
unmatched labels include punctuation-prefixed labels, apparent suffixed IDs,
and labels that cannot safely be corrected from spelling alone.
Solution
Approved by the investigator on 2026-09-18: standardize the 609 uniquely matched rows to the stored biography spelling and exclude all 6,509 NULL-focal rows and 7,495 unmatched-label rows. This excludes 14,004 rows before overlap with other problems. Do not derive a focal from coordinates, community, date, another observation, or a nearby follow.
* (#222) FOLLOW_MAP_LOCATION event times cannot be represented
Problem
PAPER and UTM events require a recorded minute from 04:00 through 20:00. The refreshed source has 25,921 NULL times, 676 times before 04:00, and 258 after 20:00. One recorded time is midnight, so midnight cannot represent NULL.
Bad Data
conversion/follow_map_location_profile.sql reports the exact classes and
representation impact. NULL times affect 25,921 PAPER and 25,914 UTM
representations; early times affect 676 UTM representations; late times
affect 7 PAPER and 258 UTM representations. Recorded seconds are always zero.
Solution
Approved by the investigator on 2026-09-18: exclude all 25,921 NULL-time rows, 676 rows before 04:00, and 258 rows after 20:00. This excludes 26,855 rows before overlap with other problems. Do not map NULL to midnight, clamp out-of-range times, round times, or derive a time from neighboring source rows.
* (#223) FOLLOW_MAP_LOCATION event keys collide
Problem
Production allows at most one PAPER and one UTM event for a location watch at a given time. Before focal normalization and after omitting NULL times, the source has 23,801 colliding PAPER focal/date/time keys and 25,561 colliding UTM keys. Generated IDs cannot make those keys representable.
Bad Data
The profile reports 47,603 PAPER rows in colliding keys, with 23,802 excess rows, and 51,255 UTM rows, with 25,694 excess rows. Only 75 PAPER keys (150 rows) and 150 UTM keys (300 rows) have one identical representation value; 23,726 PAPER keys and 25,411 UTM keys contain genuinely different recorded representations. One UTM key has 14 source rows.
The complete source additionally has 151 exact duplicate pairs across all 14 columns. A generated conversion ID does not establish that either copy is a separate production observation.
Solution
Approved by the investigator on 2026-09-18: exclude every source row whose
canonical focal/date/time key collides for either available representation.
This excludes the entire source row, rather than retaining half of a dual row,
and removes 51,457 rows before overlap with other problems. Do not add
arbitrary time offsets, select a row by generated order, or let ON CONFLICT
discard rows.
* (#224) FOLLOW_MAP_LOCATION community recovery counts depend on focal normalization
Problem
The investigator approved dated COMM_MEMBS as canonical, Unknown only for
a NULL source community without dated membership, and exclusion of unsupported
non-NULL communities and valid non-NULL communities without matching dated
membership or disagreeing with membership.
The previously approved exact counts used a case-sensitive
btrim(focal) = COMM_MEMBS.AnimID join. The loader specification instead
requires canonical focal matching by lower(normalize(btrim(value))).
Applying that rule resolves 118 additional rows through Nas to NAS, fd
to FD, pl to PL, and sw to SW.
Bad Data
On the refreshed source, canonical matching reports:
- 1,095,525 NULL-source rows with membership;
- 14,412 NULL-source rows without membership;
- 712,457 valid-source rows agreeing with membership;
- 245 valid-source rows without membership;
- 1,105 valid-source rows disagreeing with membership; and
- 22 unsupported non-NULL source communities.
The canonical result would exclude 1,372 rows rather than 1,455. The old
trim-only result remains exactly 1,095,490, 14,447, 712,374, 328, 1,105, and
22 respectively. The 83-row exclusion difference is 51 fd, 14 pl, and 18
sw rows whose valid source community agrees with canonical membership. The
other 35 newly resolved rows are NULL-community Nas rows and would change
from Unknown to membership community KK.
Solution
Canonical focal standardization was approved under Problem #221. The final Problem #224 exclusion union is therefore 1,372 rows: 22 unsupported non-NULL communities, 245 valid communities without dated membership, and 1,105 valid communities disagreeing with membership. Do not preserve the old counts by using a less canonical identity join.
(#225) LOCATION_ORIGINS lacks approved support values
Problem
LOCATION_ORIGINS is empty, but every PAPER and UTM detail requires an origin.
The source contains GPS on 1,113,996 rows, Paper Map on 15,165 rows,
paper maps on 27,273 rows, and NULL on 667,332 rows.
Bad Data
The profile reports that NULL origin affects 661,438 paper representations and 667,332 UTM representations. Of the latter, 5,894 are UTM-only rows. A UTM representation therefore does not prove GPS origin, and dual historical rows may contain coordinates converted from a paper map.
Solution
Approved by the investigator on 2026-09-18. Populate canonical origins GPS
(GPS), PAPER (Paper map), and UNKNOWN (Unknown). The uppercase code
is required by the production support-table constraint. Map source GPS to
GPS, both Paper Map and paper maps to PAPER, and SQL NULL to UNKNOWN.
Apply the same source mapping to both representations; do not infer GPS solely
from the presence of coordinates.
(#226) FOLLOW_MAP_LOCATION entered-person support is incomplete
Problem
Entered is NULL on 1,781,328 rows. Recorded values include two labels for
Danielle Lodge and the full name Julianna Turner; those values did not
previously have complete PEOPLE support.
Bad Data
The refreshed profile reports 17,794 DL, 8,902 Danielle Lodge, 7,968
Julianna Turner, 2,314 KS, 4,806 SM, and 654 SR rows. The refreshed
support table contains UNK, KS, SM, SR, DL, and Julianna Turner
with no normalized duplicate.
Solution
Approved by the investigator on 2026-09-18. Map NULL to UNK; map both DL
and Danielle Lodge to DL; map Julianna Turner to that same-named code;
and preserve KS, SM, and SR. Match with
lower(normalize(btrim(value))). The manually maintained clean.people seed
contains DL and Julianna Turner, both described as follow map location data
entry. This decision excludes no rows.
(#227) FOLLOW_MAP_LOCATION contains spatial anomalies
Problem
Thirty-nine UTM rows have coordinates outside the expected local orientation, and two rows have elevation 20,000. The production numeric limits equal the source extrema, so constraint acceptance is not evidence of geographic validity.
Bad Data
conversion/follow_map_location_profile.sql prints all 41 anomaly rows. The
39 coordinate rows include equal axes, axes both near 9.48 million, and values
that appear to have missing digits. The two elevation rows are dated
2020-03-08 (KOM) and 2021-06-06 (AP).
Solution
Approved by the investigator on 2026-09-18: preserve all 39 unusual coordinate rows and both elevation-20,000 rows exactly as recorded. They satisfy current production bounds. Do not swap axes, add digits, replace elevation by heuristic, or exclude these rows solely because they are anomalous.
(#228) FOLLOW_MAP_LOCATION has NULL optional metadata
Problem
Production FollowNum and Notes are NOT NULL but allow empty text. Source
FollowNum is NULL on 1,163,270 rows and Notes is NULL on 1,823,410 rows.
fml_update is populated on 48,796 rows but has no production destination.
Bad Data
The profile reports 18,541 distinct populated follow numbers of length 5
through 10, and 356 populated notes containing 197 distinct values. Every
populated value can be preserved exactly. Every populated fml_update is
2018-05-20.
Solution
Approved by the investigator on 2026-09-18: map SQL NULL FollowNum and
Notes to empty text while preserving populated values exactly. Retain
fml_update only in the clean audit source and do not append it to notes.
* (#229) FOLLOW_MAP_LOCATION dates exceed focal departure dates
Problem
WATCHES rejects a watch before the focal's birth or entry and after the
focal's departure from study. The source has 229 canonically matched rows
after BIOGRAPHY_DATA.DepartDate, of which 47 are already excluded by both
Problems #222 and #224. The approved Problems #220-#228 projection therefore
still contained 182 rows after departure. There are no projected rows before
birth or entry.
All 182 affected rows are UTM-only GPS records with NULL source community, NULL entered person, and valid event times. They form two location watches:
- focal
S, 134 rows on 2017-11-29 from 06:46 through 18:42, after departure
date 1968-01-19; and
- focal
DE, 48 rows on 2017-12-27 from 12:41 through 17:02, after departure
date 1974-05-07.
Bad Data
SELECT source.*,
biography_data.birthdate,
biography_data.entrydate,
biography_data.departdate
FROM clean.follow_map_location AS source
JOIN sokwedb.biography_data
ON lower(normalize(biography_data.animid)) =
lower(normalize(btrim(source.fml_fol_b_focal_animid)))
WHERE source.fml_fol_date < biography_data.birthdate
OR source.fml_fol_date < biography_data.entrydate
OR source.fml_fol_date > biography_data.departdate
ORDER BY source.fml_conversion_id;
The first full chunked load stopped when inserting the S location watch on
2017-11-29 because the WATCHES trigger reported its 1968-01-19 departure.
Solution
Approved by the investigator on 2026-09-18: exclude all post-departure rows. The predicate identifies 229 raw rows; 47 overlap both Problems #222 and #224, so Problem #229 adds exactly 182 rows to the exclusion union. The final distinct union is 93,470, leaving 1,730,296 eligible source rows. Do not weaken the production biography-date trigger, substitute another focal, or alter the recorded location dates.
(#230) Location events lack a matching focal arrival interval
Problem
Production documentation expects location times to fall within a focal's arrival interval on a B watch, but the database reports this condition as a warning rather than rejecting the event. Of 2,358,936 loaded PAPER and UTM events, 779,682 do not fall within a matching focal arrival interval.
The refreshed post-load profile separates them as follows:
- 724,075 events have no B watch for the focal/date: 196 PAPER and 723,879 UTM;
- 404 events have a B watch but no ARR interval: 202 PAPER and 202 UTM; and
- 55,203 events are outside every ARR interval on an existing B watch: 19,118
PAPER and 36,085 UTM.
These classes cover 1973-01-04 through 2022-11-28. The large no-B-watch UTM class reflects location coverage continuing well beyond the consolidated follow data; it is not evidence from which to invent a follow or arrival time.
Bad Data
Join location EVENTS and their L WATCHES rows to B watches on focal/date,
then test whether the location start falls between an ARR event's start and
stop. conversion/load_follow_map_location_validation.sql reports the exact
779,682-event total after proving source-to-target parity.
Solution
Approved by the investigator on 2026-09-18: preserve all 779,682 events as recorded. The production schema permits them and the arrival relationship is documentation guidance enforced as a warning. Validation asserts the exact count and continues to report it. Do not create B watches, manufacture ARR intervals, alter location times, or exclude otherwise eligible source rows to satisfy this guidance.
* (#231) CHIMPS_ON_TIKIS source identifiers lack biography support
Problem
Three clean.chimps_on_tikis_lookup rows cannot satisfy the required
CHIMPS_ON_TIKIS.AnimID foreign key: MGENI occurs once in KK and once in
MT, and VAN mtoto occurs once in KK. MGENI does not identify a
particular production individual. VAN mtoto is not a production AnimID.
VIS is VAN's only supported child alive on the Tiki creation and deployment
dates and had KK membership, but that evidence does not establish that the
source label denotes VIS.
Bad Data
SELECT source.*
FROM clean.chimps_on_tikis_lookup AS source
LEFT JOIN sokwedb.biography_data AS biography
ON biography.animid = source.b_animid
WHERE biography.animid IS NULL
ORDER BY source.tiki_creation_date,
source.community_id,
source.b_animid;
The dated source returns exactly:
b_animid | tiki_creation_date | community_id | tiki_deployment_date | new_addition -----------+--------------------+--------------+----------------------+------------- MGENI | 2020-08-04 | KK | 2020-08-20 | N VAN mtoto | 2020-08-04 | KK | 2020-08-20 | Y MGENI | 2020-08-04 | MT | 2020-08-20 | N
Solution
Approved by the investigator on 2026-09-19: exclude exactly these three
complete source tuples. Do not create a biography row for MGENI, infer an
individual for either MGENI row, or map VAN mtoto to VIS. The loader
sanity check must prove the exact exclusion inventory and reject any remaining
unsupported identifier. The dated source has 84 rows, leaving 81 eligible
rows.
Implemented and verified on 2026-09-19. The shared projection compares all
five source columns with IS NOT DISTINCT FROM and excludes only the approved
tuples. Read-only sanity reproduced 84 source rows, the exact three-row
exclusion multiset, and 81 supported unique projected rows. The direct load
produced 81 target rows, including 13 TRUE and 68 FALSE newcomer values;
bidirectional EXCEPT ALL returned no payload differences. An intentional
post-insert failure rolled all 81 rows back to zero. An approved clean retry
reproduced the same payload digest, 28e17175eba7622cbf9afc7130dc1900.
(#232) CHIMPS_ON_TIKIS deployment-date index targets AnimID
Problem
The index named chimps_on_tikis_tikideployementdate is defined on animid
instead of tikideploymentdate. It duplicates the separate AnimID index and
leaves the documented deployment-date lookup unindexed. The index name also
misspells "deployment".
Bad Data
SELECT indexname, indexdef
FROM pg_indexes
WHERE schemaname = 'housekeeping'
AND tablename = 'chimps_on_tikis'
AND indexname = 'chimps_on_tikis_tikideployementdate';
The live PostgreSQL 18.6 database reports an index using btree (animid).
The owning definition is in
db/schemas/housekeeping/indexes/create/chimps_on_tikis.m4; the corresponding
drop name is in the neighboring drop M4.
Solution
Approved by the investigator on 2026-09-19: before loading this table, replace
the defective index with correctly spelled
chimps_on_tikis_tikideploymentdate on (tikideploymentdate). Update both
owning M4 files, regenerate owned index SQL artifacts, and verify the live
index definition in a disposable database. This schema prerequisite excludes
no source rows.
Implemented and verified on 2026-09-19. The owning create and drop M4 files
now use the corrected index, while the drop definition also removes the legacy
misspelled name during an in-place migration. Repository generation rebuilt
the per-object and aggregate index SQL. Both the focused local migration and
the freshly rebuilt disposable integration schema reported exactly one index
on tikideploymentdate, named
chimps_on_tikis_tikideploymentdate, and no legacy index.
* (#233) SIV_STATUS_BOUT comma-bearing names shifted into later columns
Problem
Five source rows have a comma-bearing original name split across
SSB_names_used, SSB_in_bio_table, and SSB_notes. The leading name
fragment is in SSB_names_used, the trailing fragment is in
SSB_in_bio_table, and the actual biography flag 1 is in SSB_notes.
These malformed values already occur in easy.siv_status_bout; they were not
introduced by a clean-stage transformation.
Bad Data
SELECT source.*
FROM clean.siv_status_bout AS source
WHERE source.ssb_in_bio_table NOT IN ('0', '1')
ORDER BY source.ssb_b_animid, source.ssb_start_date;
The dated source returns exactly five rows: DB with fragments "Darbee and
NA"; TOF with "Tofiki and gone"; and all three TT bouts with
"Kati (Tita) and Kati-tita". Each row has SSB_notes = '1'.
Solution
Approved by the investigator on 2026-09-19: exclude these five exact complete
source tuples. The final decision supersedes an earlier consideration of
reconstructing the values in clean. Do not repair the imported fields,
infer punctuation, or use a broad identifier-only exclusion. Sanity must
prove the exact five-row multiset, and the canonical projection must compare
all ten source columns with NULL-safe equality.
Implemented and verified on 2026-09-19. conversion/siv_status_bout_projection.sql
encodes the exact five-row DB/TOF/TT tuple set, including the two
literal embedded double-quote characters recovered with quote_literal.
Read-only sanity, run against the local PostgreSQL 18.6 sokwedb database,
reproduced this exact multiset with bidirectional EXCEPT ALL and confirmed
it overlaps Problem #234 in exactly one row and Problem #235 in none.
* (#234) SIV_STATUS_BOUT required test dates are missing
Problem
Twelve source rows cannot satisfy the target's non-NULL FirstTestDate and
LastTestDate columns. All 11 unk bouts have both dates absent. The VID
positive bout has FirstTestDate = 2000-09-09 and a NULL LastTestDate.
Bad Data
SELECT source.*
FROM clean.siv_status_bout AS source
WHERE source.ssb_first_test_date IS NULL
OR source.ssb_last_test_date IS NULL
ORDER BY source.ssb_b_animid, source.ssb_start_date;
The 11 unknown bouts belong to BIM, CH100, ERI, FO, GIM, HAI,
MAK, RUD, TT, YD, and ZS. The TT unknown bout is also one of the
five Problem #233 shifted rows.
Solution
Approved by the investigator on 2026-09-19: exclude the 12 exact complete
source tuples. Do not fill absent tests from bout boundaries and do not change
target nullability. Match all ten source columns with NULL-safe equality.
Because the TT row overlaps Problem #233, count it once in the combined
exclusion set.
This decision removes every source unk bout. Ten retained positive rows
therefore keep BoutNumber = 3 without a loaded bout 2. Preserve those source
bout numbers; do not renumber or synthesize replacement bouts. Update the
production documentation to report the resulting snapshot accurately.
Implemented and verified on 2026-09-19. The projection excludes the exact
12-row tuple set, including the TT row shared with Problem #233. Sanity
reproduced 11 NULL FirstTestDate rows, 12 NULL LastTestDate rows, and
confirmed the resulting 164-row projection contains zero unk rows and
exactly 10 BoutNumber = 3 rows with no corresponding loaded
BoutNumber = 2 row, including CH100, whose eligible negative and
positive bouts are retained.
* (#235) SIV_STATUS_BOUT AMA incorrectly claims biography support
Problem
The source row for AMA has SSB_in_bio_table = '1', but no exact AMA
identifier exists in BIOGRAPHY_DATA. AME exists and the source note says
the row includes AME samples, but that is not sufficient evidence to replace
the source identifier.
Bad Data
SELECT source.*
FROM clean.siv_status_bout AS source
WHERE source.ssb_b_animid = 'AMA'
AND source.ssb_in_bio_table = '1'
AND NOT EXISTS (
SELECT 1
FROM sokwedb.biography_data AS biography
WHERE biography.animid = source.ssb_b_animid);
The dated source returns one AMA negative bout from 2014-03-20 through
2019-04-05, with tests from 2017-01-31 through 2019-04-05 and bout number 1.
Solution
Approved by the investigator on 2026-09-19: exclude this exact complete source
tuple. Do not map AMA to AME, recompute the flag, or create biography
support. Preserve all other unsupported source identifiers whose flags are
0, as the target intentionally permits identifiers outside biography.
Implemented and verified on 2026-09-19. The projection excludes the exact
AMA tuple, including its literal doubled-double-quote note text. Sanity
confirmed the 21 remaining unsupported identifiers (CH064 through CH128
and TUM) are retained with InBioTable = FALSE and that no eligible row
has an InBioTable/biography-support mismatch.
(#236) SIV_STATUS_BOUT absent notes cannot satisfy the target
Problem
The source contains 169 NULL SSB_notes values, while target Notes is
non-NULL. The target's notonlyspaces_check explicitly permits the empty
string as the representation of no note.
After the approved Problems #233 through #235 exclusion union, 158 eligible rows have NULL notes and six have non-NULL notes.
Bad Data
SELECT COUNT(*) FROM clean.siv_status_bout AS source WHERE source.ssb_notes IS NULL;
The dated source returns 169.
Solution
Approved by the investigator on 2026-09-19: map an eligible NULL note to the
empty string with COALESCE(source.ssb_notes, ). Preserve every non-NULL
note exactly. Sanity and validation must prove that this is the only notes
transformation and that no projected note contains only spaces.
Problems #233, #234, and #235 contain 5, 12, and 1 rows respectively. One
TT row overlaps #233 and #234, so their union excludes 17 distinct rows from
181 and leaves 164 rows for conversion.
Implemented and verified on 2026-09-19. COALESCE(source.ssb_notes, ) is
the projection's only notes transformation. Read-only sanity and validation
against the local PostgreSQL 18.6 sokwedb database confirmed 158 loaded
empty notes came only from NULL source notes, all 6 non-NULL notes are
preserved byte-for-byte, and no loaded note contains only spaces.
The direct set-based loader
(conversion/load_siv_status_bout.sql) inserted exactly 164 rows: 132 neg,
32 pos, 0 unk; 154 BoutNumber = 1, 0 BoutNumber = 2, 10
BoutNumber = 3; 142 TRUE and 22 FALSE InBioTable values; and 158
empty/6 nonempty notes. Bidirectional EXCEPT ALL between the canonical
projection and the loaded rows returned no differences in either direction.
An intentional post-insert failure (SELECT 1/0) rolled the single
transaction back to zero target rows. An authorized TRUNCATE of only the
disposable target table followed by an unchanged sanity/load/validation
sequence reproduced the identical sorted ten-column payload digest,
8b243c9b7f53d79f9fa1eae640e26b51, on both runs. git diff --check, a
local-only Make dry run of the full load_data sequence, and a direct m4
and repository doc-build render of doc/src/housekeeping/siv_status_bout.m4
all passed without error.
(#237) SUBADULT_ARRIVALS_LOG FirstTikiDate is declared NOT NULL contrary to its own documentation
Problem
db/schemas/housekeeping/tables/create/subadult_arrivals_log.m4 declares
firsttikidate DATE NOT NULL, but the owning documentation, added in the
same commit, says the column "may be NULL when there is no information on
the date of permanent addition." 156 of 209 source rows (75%) have a NULL
SA_first_tiki_date, each carrying an explanatory note describing why no
discrete tiki date applies. A NULL FirstTikiDate is this table's normal,
permanent representation for most individuals, not missing data awaiting
entry.
Bad Data
SELECT count(*) FILTER (WHERE sa_first_tiki_date IS NULL) AS null_date
, count(*) FILTER (WHERE sa_first_tiki_date IS NOT NULL) AS dated
FROM clean.subadult_arrivals_log;
The dated source returns 156 NULL and 53 non-NULL rows. git show cc3640b
--stat shows the table, its documentation, its indexes, and its trigger
were all added in one commit, so this contradiction has existed since the
table was created and was never exercised by a loader.
Solution
Approved by the investigator on 2026-09-19: correct the schema defect
rather than exclude the 156 legitimately NULL-dated rows. Remove NOT
NULL from the firsttikidate column in the owning create M4, regenerate
the owned table SQL artifact, and verify the corrected nullability in a
disposable or migrated database before loading. Leave the unique AnimID
index, the FirstTikiDate index, the AnimID foreign key and NONE
check, and the update trigger unchanged. This is a target-schema
correction, not a clean-stage mutation, and excludes no source rows on its
own.
Implemented and verified on 2026-09-19. NOT NULL was removed from
firsttikidate in
db/schemas/housekeeping/tables/create/subadult_arrivals_log.m4; the owned
table SQL artifact was regenerated with make db/schemas/createtables.sql
and shows firsttikidate DATE with no other column or constraint changed.
ALTER TABLE housekeeping.subadult_arrivals_log ALTER COLUMN firsttikidate
DROP NOT NULL was applied to the local PostgreSQL 18.6 sokwedb disposable
database, and information_schema.columns confirmed firsttikidate is
nullable while id, animid, and notes remain NOT NULL and the unique
AnimID index, FirstTikiDate index, AnimID foreign key and NONE
check, and update trigger are unchanged.
* (#238) SUBADULT_ARRIVALS_LOG CT's FirstTikiDate follows CT's own departure date
Problem
CT's parsed FirstTikiDate (1986-03-01) is after CT's own
BIOGRAPHY_DATA.DepartDate (1985-09-30). The target has no constraint
tying FirstTikiDate to biography dates, so this row would load without
error, but a tiki date recorded after an individual's departure is not
trustworthy evidence of when that individual was added to the tiki sheets.
Bad Data
SELECT source.*, biography.departdate
FROM clean.subadult_arrivals_log AS source
JOIN sokwedb.biography_data AS biography
ON biography.animid = source.sa_b_animid
WHERE to_date(source.sa_first_tiki_date, 'MM/DD/YY') > biography.departdate;
The dated source returns exactly one row: CT, 03/01/86, with note "Btwn
DOB and CA death, arrivals same as CA; Elo confirmed that always accounted
for on tikis afterwards." No other row falls outside its individual's
birth/entry/departure window.
Solution
Approved by the investigator on 2026-09-19: exclude this exact row rather
than preserve it as a documented semantic anomaly. Match on the exact
source tuple (sa_b_animid, sa_first_tiki_date, sa_notes) with
NULL-safe equality; do not exclude by AnimID alone or use a broad
date-comparison predicate as the operative exclusion, and do not infer a
corrected date. This is the only approved exclusion; it leaves 208
eligible rows.
Implemented and verified on 2026-09-19.
conversion/subadult_arrivals_log_projection.sql encodes the exact
one-row CT tuple. Read-only sanity, run against the local PostgreSQL
18.6 sokwedb database, reproduced this exact tuple with bidirectional
EXCEPT ALL and confirmed no other eligible row falls outside its
individual's birth/entry/departure window.
(#239) SUBADULT_ARRIVALS_LOG absent notes cannot satisfy the target
Problem
The source contains 6 NULL SA_notes values, all outside the Problem #238
exclusion. The target Notes is NOT NULL but explicitly permits the empty
string.
Bad Data
SELECT count(*) FROM clean.subadult_arrivals_log WHERE sa_notes IS NULL;
The dated source returns 6.
Solution
Approved by the investigator on 2026-09-19: map an eligible NULL note to
the empty string with COALESCE(source.sa_notes, ). Preserve every
non-NULL note exactly, including two rows (FAD, KEA) that carry leading
or trailing spaces around substantive text. Sanity and validation must
prove that this is the only notes transformation and that no projected
note contains only spaces. This mapping, together with Problem #237's
schema fix and Problem #238's exclusion, leaves 208 rows for conversion:
209 source rows minus the one Problem #238 exclusion.
Implemented and verified on 2026-09-19.
COALESCE(source.sa_notes, ) is the projection's only notes
transformation. Read-only sanity and validation against the local
PostgreSQL 18.6 sokwedb database confirmed 6 loaded empty notes came
only from NULL source notes, all 202 non-NULL notes are preserved
byte-for-byte (including the two rows with leading/trailing spaces), and
no loaded note contains only spaces.
The direct set-based loader (conversion/load_subadult_arrivals_log.sql)
inserted exactly 208 rows: 156 NULL and 52 non-NULL FirstTikiDate
values (extent 1976-05-01 through 2013-09-22, unchanged by the CT
exclusion), and 6 empty/202 nonempty notes. Every non-NULL FirstTikiDate
was proven to round-trip through to_char(..., 'MM/DD/YY') back to its
exact source text. Bidirectional EXCEPT ALL between the canonical
projection and the loaded rows returned no differences in either
direction. An intentional post-insert failure (SELECT 1/0) rolled the
single transaction back to zero target rows. An authorized TRUNCATE of
only the disposable target table followed by an unchanged
sanity/load/validation sequence reproduced the identical sorted
four-column payload digest, 6eac68b10ebb802e3c78b9109e28d3d7, on both
runs. git diff --check, a local-only Make dry run of the full
load_data sequence, and a direct m4 and repository doc-build render
of doc/src/housekeeping/subadult_arrivals_log.m4 all passed without
error.
(#240) TLK_BRECORD_NOTES_CODES two Abbreviation values carry a trailing space
Problem
"ID" 94's "Abbreviation" is l (trailing space) with "Meaning" of
laugh; "ID" 186's "Abbreviation" is th (trailing space) with
"Meaning" of threaten. Both violate the target's
trimmedofspaces_check. Trimming either value makes it collide with an
existing unpadded row ("ID" 93 l -> locomotion/travel; "ID" 185
th -> throw), folding directly into the Problem #241 duplicate-
abbreviation resolution.
Bad Data
SELECT "ID", "Abbreviation", "Meaning"
FROM clean.tlk_brecord_notes_codes
WHERE "Abbreviation" IS DISTINCT FROM BTRIM("Abbreviation");
The dated source returns exactly two rows: 94, "l ", laugh and 186,
"th ", threaten.
Solution
Approved by the investigator on 2026-09-19, as an explicit exception to
this repository's usual prohibition on clean-stage mutation: trim
"Abbreviation" in conversion/clean.sql, in its own reviewable
statement, rather than in the loader or projection. An initial
implementation matched the UPDATE by exact "ID" IN (94, 186); on
review the investigator preferred applying BTRIM unconditionally, as a
general clean-stage normalization, since it produces the identical result
here (no other row is padded) and reads more simply as ordinary cleanup
rather than a two-row special case. The dated source affects exactly the
same two rows either way, "ID" 94 and "ID" 186. After this fix, "ID"
94's abbreviation is l and "ID" 186's is th, each now a member of an
ordinary Problem #241 duplicate group.
Implemented and verified on 2026-09-19. Added an unconditional
UPDATE ... SET "Abbreviation" = BTRIM("Abbreviation") to
conversion/clean.sql, independently reviewable from the loader, with a
comment documenting that the dated source's only two affected rows are
"ID" 94 and "ID" 186. Read-only re-checks against the local
PostgreSQL 18.6 sokwedb database confirmed the two exact pre-fix rows
before the trim, that applying the statement leaves zero padded rows
afterward, and that the two documented rows' post-fix unpadded values
(l, th) are unchanged from what the narrowly-targeted form would have
produced.
* (#241) TLK_BRECORD_NOTES_CODES duplicate abbreviations after trimming
Problem
After Problem #240's fix, 26 case-insensitive abbreviations occur more than
once, with genuinely different Meaning text recorded for the same
abbreviation depending on note-taking context (for example r means
rest, reassurance, reunion, or resume across four separate rows).
The target permits only one row per case-insensitive abbreviation via a
unique index on lower(normalize(Abbreviation)).
Bad Data
SELECT lower(normalize(BTRIM("Abbreviation"))) AS norm_abbr, count(*)
, array_agg("ID" ORDER BY "ID") AS ids
, array_agg("Meaning" ORDER BY "ID") AS meanings
FROM clean.tlk_brecord_notes_codes
GROUP BY 1
HAVING count(*) > 1
ORDER BY 1;
The dated source returns exactly 26 groups covering 55 rows: 24 groups of 2
rows, one group of 3 (ds), and one group of 4 (r).
Solution
Superseded on 2026-09-19. An initial approval picked the lowest source
"ID" in each group to survive. On review the investigator rejected that
tie-break: choosing a survivor's Meaning over its siblings' is itself an
uncorroborated judgment call about which recorded meaning is "correct",
which is exactly what a targeted, documented exclusion is meant to avoid.
Approved by the investigator on 2026-09-19 (final): exclude every row in
each of the 26 case-insensitive duplicate-abbreviation groups, not only the
non-survivors. None of the 26 ambiguous abbreviations is loaded under any
Meaning; only the 155 rows whose abbreviation is unique in the source
(after Problem #240's trim) are eligible. This drops all 55 grouped rows,
not 29. The excluded rows, and their meanings, remain visible, unmodified,
in clean.tlk_brecord_notes_codes. Do not infer, rank, or merge meanings;
do not pick a survivor by "ID" or any other criterion.
Implemented and verified on 2026-09-19. conversion/tlk_brecord_notes_codes_projection.sql
excludes every row whose case-insensitive abbreviation occurs more than
once in clean.tlk_brecord_notes_codes (after the Problem #240 trim),
proved against an independently-derived 55-row, 26-abbreviation exclusion
set. Read-only sanity (conversion/load_tlk_brecord_notes_codes_sanity.sql)
against the local PostgreSQL 18.6 sokwedb database confirmed the
exclusion set matches exactly and that the canonical projection is exactly
155 rows with 155 distinct case-insensitive abbreviations, none of them one
of the 26 excluded abbreviations.
The direct set-based loader (conversion/load_tlk_brecord_notes_codes.sql)
inserted exactly 155 rows. Bidirectional EXCEPT ALL between the
canonical projection and the loaded rows
(conversion/load_tlk_brecord_notes_codes_validation.sql) returned no
differences in either direction, every loaded abbreviation was proven
unique under lower(normalize(...)) and to have exactly one matching
source row, every loaded Meaning was proven to match that single source
row exactly, and none of the 26 excluded abbreviations was proven present
in the loaded rows. An intentional post-insert failure (SELECT 1/0)
rolled the single transaction back to zero target rows. An authorized
TRUNCATE of only the disposable target table followed by an unchanged
sanity/load/validation sequence, run twice, reproduced the identical
sorted two-column payload digest, 27da98a74e6378218418652dda914c95, on
both runs. git diff --check, a
local-only Make dry run of the full load_data sequence confirming correct
ordering, a direct Make-driven execution of the new
load_tlk_brecord_notes_codes_sanity/load_tlk_brecord_notes_codes/
load_tlk_brecord_notes_codes_validation targets, and a direct m4 render
of doc/src/housekeeping/tlk_brecord_notes_codes.m4 all passed without
error.
* (#242) FOOD_VARIATIONS eligible rows lack a LocalName
Problem
Problem #161 deferred all of clean.food_variations_lookup to a future
conversion because of missing and unresolved LocalName values. Picking
this conversion back up: the target housekeeping.food_variations.LocalName
is NOT NULL, but 224 of the source's 1,575 rows have a NULL
fvl_fl_local_food_name. 40 of those 224 rows have a
fvl_food_spelling_variant that itself exactly matches an existing
codes.food_names entry (for example DONGO, KIFUMUFUMU), but the
other 184 do not, so there is no single reliable substitute value across
all 224 rows.
Bad Data
SELECT count(*) AS null_localname_rows
, count(*) FILTER (
WHERE EXISTS (
SELECT 1 FROM codes.food_names AS fn
WHERE lower(normalize(fn.name)) =
lower(normalize(fvl.fvl_food_spelling_variant))))
AS variant_matches_food_names
FROM clean.food_variations_lookup AS fvl
WHERE fvl.fvl_fl_local_food_name IS NULL;
The 2026-09-19 snapshot returns 224 and 40.
Solution
Approved by the investigator on 2026-09-19: exclude all 224 rows with a
NULL LocalName, including the 40 whose Variant happens to match a
codes.food_names entry. Do not infer a LocalName from the Variant,
even for those 40 rows. This is a straightforward domain predicate
(fvl_fl_local_food_name IS NOT NULL), not an exact-tuple exclusion list,
since excluding by this condition cannot ambiguously over- or
under-exclude the way a duplicate-abbreviation tie-break could. This
leaves at most 1,351 eligible rows before any other approved exclusion.
Implemented and verified on 2026-09-20. conversion/food_variations_projection.sql
excludes every row via the domain predicate fvl_fl_local_food_name IS
NOT NULL. Read-only sanity (conversion/load_food_variations_sanity.sql)
against the local PostgreSQL 18.6 sokwedb database confirmed the exact
224-row NULL-LocalName count and that the canonical projection is
exactly 1,351 rows with 1,351 distinct case-insensitive Variant values.
The direct set-based loader (conversion/load_food_variations.sql)
inserted exactly 1,351 rows. Bidirectional EXCEPT ALL between the
canonical projection and the loaded rows
(conversion/load_food_variations_validation.sql) returned no
differences in either direction, every loaded Variant was proven unique
under lower(normalize(...)), every loaded LocalName was proven to
match its source row exactly, and no loaded row was proven to have a NULL
source LocalName. An intentional post-insert failure (SELECT 1/0)
rolled the single transaction back to zero target rows. An authorized
TRUNCATE of only the disposable target table followed by an unchanged
sanity/load/validation sequence, run twice, reproduced the identical
sorted two-column payload digest, c510323457dcb801412441eda0ceb09c, on
both runs. git diff --check, a local-only Make dry run of the full
load_data sequence confirming correct ordering, a direct Make-driven
execution of the new load_food_variations_sanity/load_food_variations/
load_food_variations_validation targets, and a direct m4 render of
doc/src/housekeeping/food_variations.m4 all passed without error.
(#243) FOOD_VARIATIONS orphaned LocalName values are not cross-checked
Problem
Of the 1,351 rows surviving Problem #242, 38 have a LocalName that does
not match any clean.food_lookup entry case-insensitively, and a
stricter 79 do not match the smaller, already-loaded production
codes.food_names table. housekeeping.food_variations declares no
foreign key to either table, so nothing in the schema forces a
resolution; several of the mismatches form typo clusters pointing at a
name that is not itself canonical either (for example eight variants —
MBOBOGOLO, MTOMBOGOLO, MUTBOGORO, MUTOBOGALO, MUTOBOGO,
MUTOBOLO, MUTOBORGORO, MUTOLGURO — all record LocalName
MTOBOGOLO, which appears in neither reference table).
Bad Data
SELECT count(*) FILTER (
WHERE NOT EXISTS (
SELECT 1 FROM clean.food_lookup AS fl
WHERE lower(fl.fl_local_food_name) = lower(fvl.fvl_fl_local_food_name)))
AS orphan_vs_food_lookup
, count(*) FILTER (
WHERE NOT EXISTS (
SELECT 1 FROM codes.food_names AS fn
WHERE lower(normalize(fn.name)) =
lower(normalize(fvl.fvl_fl_local_food_name))))
AS orphan_vs_food_names
FROM clean.food_variations_lookup AS fvl
WHERE fvl.fvl_fl_local_food_name IS NOT NULL;
The 2026-09-20 snapshot returns 38 and 79, unchanged since 2026-09-19.
Solution
Approved by the investigator on 2026-09-20: do not cross-check LocalName
against clean.food_lookup, codes.food_names, or any other table for
this conversion. Load every eligible row's LocalName as free text
exactly as recorded, matching, mismatching, or orphaned alike. Sanity may
report the 38/79 counts as informational profiling, but neither figure
excludes any row. This mapping excludes no additional rows beyond
Problem #242's 224.
Implemented and verified on 2026-09-20. The canonical projection applies
no cross-check against clean.food_lookup or codes.food_names; sanity
reports the 38/79 mismatch counts as RAISE NOTICE profiling only.
Validation confirmed every loaded LocalName matches its source row
exactly regardless of whether it matches either reference table, and the
1,351-row loaded count matches 1,575 source - 224 Problem #242
exclusions exactly, confirming no additional row was excluded under this
resolution.
(#244) REPRO_STATES cannot represent adolescent and not-seen days
Problem
clean.female_reproductive_states.repro_state contains A for adolescent
and SQL NULL for not seen, but production REPRO_STATES.State is NOT NULL
and permits only cycling (C), pregnant (P), and lactating (L). The
dated 2026-09-19 source has 7,582 A rows and 218,370 NULL rows.
Bad Data
SELECT repro_state, count(*) FROM clean.female_reproductive_states GROUP BY repro_state ORDER BY repro_state NULLS LAST;
Solution
Approved by the investigator and implemented on 2026-09-21: adolescent A
was added to the target state domain and REPRO_STATES.State was made
nullable. Map source A, C, P, and LA to target A, C, P, and
L; preserve source NULL as target NULL. Do not carry a previous state
into a not-seen day. The generated SQL and documentation were regenerated;
catalog inspection and a rollback-only NULL-State insert verified the change.
(#245) REPRO_STATES requires sparse support attributes
Problem
Production Source, EstrusDayCert, and LactEndCert are NOT NULL, but
the dated source contains 646,257, 626,192, and 646,602 NULL values
respectively. The 30 rows having an EstrusDay but NULL EstrusDayCert are
valid under the approved sparse representation.
Bad Data
SELECT count(*) FILTER (WHERE state_change_dec_source IS NULL) AS null_source
, count(*) FILTER (WHERE day_of_est_cert IS NULL) AS null_ed_cert
, count(*) FILTER (WHERE lactat_end_cert IS NULL) AS null_le_cert
, count(*) FILTER (
WHERE day_of_estr IS NOT NULL AND day_of_est_cert IS NULL)
AS estrus_day_without_cert
FROM clean.female_reproductive_states;
Solution
Approved by the investigator and implemented on 2026-09-21: the three production columns were made nullable. Preserve source NULL as SQL NULL; do not carry values forward and do not convert NULL to empty text. The support tables reject empty keys, and an empty key would invent a category rather than represent missing information. Catalog inspection and a rollback-only sparse-row insert verified the change.
(#246) REPRO_STATES support tables are empty
Problem
The four required support tables are empty. The source has twelve distinct
trimmed change-source labels, ED certainty values 1 through 5 plus the
Problem #247 typo, LE certainty values 1 through 4, and three parity
values. The Access dump contains no lookup tables defining them.
Bad Data
SELECT upper(btrim(state_change_dec_source)) AS code, count(*) FROM clean.female_reproductive_states WHERE state_change_dec_source IS NOT NULL GROUP BY 1 ORDER BY 1; SELECT parity, count(*) FROM clean.female_reproductive_states GROUP BY parity ORDER BY parity;
Solution
Approved by the investigator and implemented on 2026-09-21: load uppercase
parity keys IMMATURE, NULLIPAROUS, and PAROUS. Normalize populated
change-source keys with upper(btrim(...)), retaining the twelve semantic
labels and expanding their state prefixes in Description; the exact
key/description pairs are recorded in repro_states_loader_handoff.md.
Load ED keys 1 through 4 with the approved descriptions based on the
length of the unobserved gap following the last full swelling. Load LE keys
1 through 4 with the approved minimum/maximum swelling, 14-day, and
first-cycle criteria. ED code 5 has no support row; see Problem #248, which
now corrects 5 to 4 in the clean schema rather than excluding it, so no
row ever carries an unsupported code. No support row represents NULL; see
Problem #245. Sanity verifies bidirectional parity of every support key and
description; the clean retry loaded exactly 12 change sources, four ED
certainties, four LE certainties, and three parities.
(#247) FEMALE_REPRODUCTIVE_STATES has estrus certainty 22
Problem
One row, SIF on 2004-03-21, has day_of_est_cert = '22'. The investigator
confirmed that this is a typo for 2.
Bad Data
SELECT * FROM clean.female_reproductive_states WHERE day_of_est_cert = '22';
Solution
Approved by the investigator and implemented on 2026-09-21: change 22 to
2 in the clean schema. The focused clean-stage execution changed exactly
one row; sanity verifies that 22 is absent and the corrected 2 count is
8,251.
(#248) FEMALE_REPRODUCTIVE_STATES estrus certainty 5 is unsupported
Problem
The dated source has 30 rows with day_of_est_cert = '5'. The investigator
provided meanings for ED certainty codes 1 through 4; code 5 has no
supported meaning.
Bad Data
SELECT * FROM clean.female_reproductive_states WHERE day_of_est_cert = '5' ORDER BY id, date;
Solution
Approved by the investigator and implemented on 2026-09-21: temporarily
exclude exactly these 30 rows from the reproductive-state projection. Code
5 is absent from ED_CERTAINTIES. Sanity verifies both the 30-row class and
that it overlaps none of Problems #251 or #254.
Amended by the investigator on 2026-09-24: convert day_of_est_cert = '5' to
4 in the clean schema instead of excluding the 30 rows. No 5 value is
retained, no row is excluded, and no support row is added for 5; the
correction is applied alongside the Problem #247 typo correction in
conversion/clean.sql. Sanity now asserts an exact 2,325-row 4 count and
rejects any remaining 5 value as a corrected-domain violation. The
approved projection grew from 649,495 to 649,525 rows, and the pre-expansion
eligible/exclusion counts changed accordingly; the refreshed counts are
verified in conversion/repro_states_profile.sql,
conversion/load_repro_states_sanity.sql, and
conversion/load_repro_states_validation.sql.
(#249) FEMALE_REPRODUCTIVE_STATES swelling bounds have no target
Problem
Source columns min_sw and max_sw have no REPRO_STATES destination. The
dated source has 160,268 rows where both are populated and 486,621 where both
are NULL.
Bad Data
SELECT min_sw, max_sw, count(*) FROM clean.female_reproductive_states GROUP BY min_sw, max_sw ORDER BY min_sw NULLS LAST, max_sw NULLS LAST;
Solution
Approved by the investigator and implemented on 2026-09-21: do not map or
convert these fields because they are generated by build_swelling_states().
They remain unchanged in clean as audit evidence and are absent from the
shared production projection. This decision excludes no rows.
(#250) FEMALE_REPRODUCTIVE_STATES has two incorrect female IDs
Problem
Female IDs OBE and UWE have no production biography match, affecting
1,404 and 29 rows respectively.
Bad Data
SELECT id, count(*)
FROM clean.female_reproductive_states
WHERE id IN ('OBE', 'UWE')
GROUP BY id
ORDER BY id;
Solution
Approved by the investigator and implemented on 2026-09-21: correct OBE to
POR and UWE to MGF in the clean schema. The focused clean-stage
execution changed exactly 1,433 rows; sanity verifies 1,404 POR and 29 MGF
rows, no old labels, and applies study-boundary checks afterward.
* (#251) FEMALE_REPRODUCTIVE_STATES dates fall outside study intervals
Problem
After applying Problem #250's female corrections, 2,145 dated source rows
fall before EntryDate or after DepartDate. This replaces the earlier
matched-ID-only count of 1,175: correcting OBE to POR adds 677
before-entry and 293 after-departure rows. The other corrected ID, MGF,
adds none.
Bad Data
WITH corrected AS (
SELECT source.*
, CASE source.id
WHEN 'OBE' THEN 'POR'
WHEN 'UWE' THEN 'MGF'
ELSE source.id
END AS animid
FROM clean.female_reproductive_states AS source)
SELECT corrected.*
FROM corrected
JOIN sokwedb.biography_data
ON biography_data.animid = corrected.animid
WHERE corrected.date < biography_data.entrydate
OR corrected.date > biography_data.departdate
ORDER BY corrected.animid, corrected.date;
Solution
Approved by the investigator and implemented on 2026-09-21: temporarily exclude every corrected-female/date row outside the inclusive study interval. The shared projection does not alter dates, and sanity independently verifies exactly 2,145 exclusions and no overlap with Problems #248 or #254.
(#252) FEMALE_REPRODUCTIVE_STATES compound offspring IDs represent twins
Problem
Five slash-delimited young_kid_id values represent twins rather than invalid
single identifiers. A single YKID foreign key cannot store both offspring.
Bad Data
SELECT young_kid_id, count(*) FROM clean.female_reproductive_states WHERE young_kid_id LIKE '%/%' GROUP BY young_kid_id ORDER BY young_kid_id;
The raw dated counts are GAB2/GAB3 6, GLI/GLD 2,013, GY/GL 296,
SG/CT 3,472, and SHO/ROO 368. After preceding exclusions, 5,546 source
rows are eligible and expand to 11,092 target rows; Problem #251 removes 609
of the SG/CT rows first.
Solution
Approved by the investigator and implemented on 2026-09-21: split each eligible compound row into two target rows, one per offspring, copying all other source payload before deriving YKAge separately for each offspring. The projection names only the five approved labels and fails sanity on any new slash-delimited label. It expands 5,546 eligible source rows to 11,092 output rows. See Problem #258 for the resulting female/date/YKID invariant.
(#253) FEMALE_REPRODUCTIVE_STATES offspring AMA is incorrect
Problem
The unmatched offspring label AMA occurs on 287 dated source rows. The
investigator confirmed that these values should identify AME.
Bad Data
SELECT * FROM clean.female_reproductive_states WHERE young_kid_id = 'AMA' ORDER BY id, date;
Solution
Approved by the investigator and implemented on 2026-09-21: correct AMA to
AME in the clean schema. The focused clean-stage execution changed exactly
287 rows, and sanity verifies that AMA is absent and 287 rows use AME.
This correction excludes no rows.
* (#254) FEMALE_REPRODUCTIVE_STATES offspring TTB2 is unsupported
Problem
The unmatched offspring label TTB2 occurs on 546 dated source rows and has
no approved production biography identity.
Bad Data
SELECT * FROM clean.female_reproductive_states WHERE young_kid_id = 'TTB2' ORDER BY id, date;
Solution
Approved by the investigator and implemented on 2026-09-21: temporarily exclude exactly these 546 rows. The projection creates no biography row and infers no replacement offspring. Sanity verifies the exact class and that it overlaps neither Problem #248 nor Problem #251. Problem #248's 2026-09-24 amendment converts its 30 rows to a supported code rather than excluding them, so this class no longer shares an exclusion mechanism with Problem
- 248, but the two remain non-overlapping in the source data.
* (#255) Source youngest-offspring ages disagree with biography
Problem
Of 268,336 originally resolved source rows, 115,527 source
youngest_kid_age values differ from Date - BIOGRAPHY_DATA.BirthDate.
After approved corrections, preceding exclusions, and twin expansion, every
retained YKID has a biography BirthDate, but 219 nonsplit rows derive a
negative age from -41 through -1. Production YKAge must be nonnegative.
Bad Data
WITH expanded AS (
SELECT source.id, source.date
, CASE btrim(part.ykid) WHEN 'AMA' THEN 'AME'
ELSE btrim(part.ykid) END AS ykid
FROM clean.female_reproductive_states AS source
CROSS JOIN LATERAL
regexp_split_to_table(source.young_kid_id, '/') AS part(ykid)
WHERE source.young_kid_id IS NOT NULL
AND source.young_kid_id <> 'TTB2')
SELECT expanded.*, biography_data.birthdate
, expanded.date - biography_data.birthdate AS derived_age
FROM expanded
JOIN sokwedb.biography_data
ON biography_data.animid = expanded.ykid
WHERE expanded.date - biography_data.birthdate < 0
ORDER BY expanded.id, expanded.date, expanded.ykid;
The implementation profile must additionally apply Problems #248, #250, and
- 251 in projection order and assert the final 219-row count independently.
Solution
Approved by the investigator and implemented on 2026-09-21: ignore source
youngest_kid_age and derive target YKAge from target Date minus the resolved
offspring's biography BirthDate. The projection excludes the 219 rows whose
derived age is negative. Sanity verifies zero unresolved retained offspring,
and validation checks every loaded age against biography. The original
115,527-value disagreement remains in the reproducible profile.
* (#256) FEMALE_REPRO_HISTORY is outside reproductive-state scope
Problem
clean.female_repro_history is related historical data but is neither a
duplicate nor a drop-in supplement for clean.female_reproductive_states.
It has 128,306 rows, including 764 female/date keys absent from the requested
daily source, and overlapping fields differ extensively.
Bad Data
SELECT count(*) AS rows
, count(DISTINCT (id, date)) AS female_date_keys
FROM clean.female_repro_history;
Solution
Approved by the investigator on 2026-09-21: do not include
clean.female_repro_history in this conversion. Retain the reconciliation
queries as profile evidence only. This decision does not exclude any row from
the requested source.
(#257) REPRO_STATES AnimID updates bypass integrity checks
Problem
repro_states_func() checks sex only on INSERT and checks study boundaries
only on INSERT or a Date change. A rollback-only probe changed only AnimID on
a valid female row to male AL on 2004-11-19, after AL's 1999-01-13
DepartDate; the UPDATE was accepted.
Bad Data
SELECT pg_get_functiondef('sokwedb.repro_states_func()'::regprocedure);
The returned function gates the sex check on TG_OP = 'INSERT' and the date
check on TG_OP = 'INSERT' OR NEW.date <> OLD.date.
Solution
Approved by the investigator and implemented on 2026-09-21: the owning M4 reruns female-sex validation whenever AnimID changes and reruns study-boundary validation whenever AnimID or Date changes. Rollback-only tests verified that updates to a male AnimID, to a female outside her study interval, and to a date outside the current female's study interval are rejected.
(#258) REPRO_STATES lacks the female/date/offspring invariant
Problem
Production permits duplicate reproductive-state rows. The earlier proposed female/date uniqueness rule is too strict because Problem #252 requires one row per twin; the intended identity is female, date, and youngest offspring. Rows with no youngest offspring must still be unique per female/date.
Bad Data
SELECT animid, date, ykid, count(*) FROM sokwedb.repro_states GROUP BY animid, date, ykid HAVING count(*) > 1 ORDER BY animid, date, ykid NULLS FIRST;
A rollback-only probe inserted two otherwise valid rows for AND on
2004-11-19 and both were accepted. The dated approved projection has zero
duplicate (AnimID, Date, YKID) keys when NULL YKID is treated as equal.
Solution
Approved by the investigator and implemented on 2026-09-21: a unique index
was added on (AnimID, Date, YKID) NULLS NOT DISTINCT, and the table
documentation describes one row per female/date/youngest-offspring.
Rollback-only tests verified that duplicate NULL and non-NULL YKID keys are
rejected while two distinct twin YKIDs on one female/date are accepted.
(#259) ELO_RANKS_DAILY_KK_F has rows outside GK's and LB's recorded community
Problem
clean.elo_ranks_daily___kk_females maps directly to sokwedb.elo_ranks_daily_kk_f,
a table scoped to the KK (Kasekela) community. An inclusive join to
COMM_MEMBS found 821 rows for four females that do not fall within a KK
interval on their ranked date:
GK, 181 rows, 1972-11-01 through 1973-04-30: 61 days have noCOMM_MEMBS
interval at all, and the remaining 120 days fall within an HK (Kahama)
interval.
LB, 638 rows, 1973-01-01 through 1974-09-30: all 638 days fall within an
HK interval.
ML, 1 row, 1986-10-24: noCOMM_MEMBSinterval (see Problem #260).AL, 1 row, 1999-01-14, insokwedb.elo_ranks_daily_kk_m: noCOMM_MEMBS
interval (see Problem #260).
This spans the historical KK/HK community split (HK starts 1973-01-01,
ends 1977-12-31 per codes.COMM_IDS). Neither the target schema nor the
current loader constrains ranking dates to a COMM_MEMBS interval, so nothing
in production would reject any treatment of these rows.
Bad Data
SELECT e.date, e.individual
FROM clean.elo_ranks_daily___kk_females e
WHERE NOT EXISTS (
SELECT 1 FROM sokwedb.comm_membs cm
WHERE cm.animid = e.individual
AND e.date BETWEEN cm.startdate AND cm.enddate
AND cm.commid = 'KK'
)
ORDER BY e.date;
Returns 820 rows (GK, LB, ML); the equivalent query against
clean.elo_ranks_daily___kk_males returns 1 row (AL).
clean.elo_ranks_daily___mt_females has zero such rows.
Solution
Approved by the investigator on 2026-09-22: preserve all 821 rows unchanged.
The daily Elo ranking is a precomputed, published output; the conversion's
role is to carry it forward as supplied, not to re-scope it by community
after the fact. No row is excluded and no rank is recalculated. This
decision applies identically to the 758 days recorded in HK and the 63 days
with no COMM_MEMBS interval at all.
(#260) ELO_RANKS_DAILY ranking dates trail DepartDate by one day for ML and AL
Problem
Two rows are dated exactly one day after the individual's
BIOGRAPHY_DATA.DepartDate:
MLinclean.elo_ranks_daily___kk_females: ranked 1986-10-24, DepartDate
1986-10-23.
ALinclean.elo_ranks_daily___kk_males: ranked 1999-01-14, DepartDate
1999-01-13.
Neither row is otherwise malformed, and the target schema does not constrain
ranking dates to an individual's study interval, so both would load without
error. Both rows are also members of the 63 "no COMM_MEMBS interval" rows
described in Problem #259.
Bad Data
SELECT e.date, e.individual, b.departdate FROM clean.elo_ranks_daily___kk_females e JOIN sokwedb.biography_data b ON b.animid = e.individual WHERE e.date = b.departdate + 1; -- and the equivalent query against clean.elo_ranks_daily___kk_males
Returns exactly the ML and AL rows above; no other individual in any of
the three sources is ranked after their own DepartDate.
Solution
Approved by the investigator on 2026-09-22: preserve both rows unchanged, as
a final recorded ranking day. Because each ranked day is a complete 1..N
permutation, excluding either individual's single row would leave that day
looking complete while no longer being one, so no partial-day exclusion is
created for either row. This decision was made together with, and is
consistent with, Problem #259's community-membership disposition, without
treating the two as the same underlying question.
(#261) ELO_RANKS_DAILY documentation misstates the ExpNumBeaten formula
Problem
The shared documentation macro (doc/include/elo_ranks_daily_macro.m4)
defines ExpNumBeaten as EloCardinal multiplied by the number of
individuals ranked that day. Every row in all three sources contradicts that
formula.
Bad Data
SELECT e.date, e.individual, e.expnumbeaten, e.elocardinal, d.n,
e.elocardinal * d.n AS documented_formula,
e.elocardinal * (d.n - 1) AS observed_formula
FROM clean.elo_ranks_daily___kk_females e
JOIN (SELECT date, count(*) AS n
FROM clean.elo_ranks_daily___kk_females
GROUP BY date) d ON d.date = e.date
ORDER BY abs(e.expnumbeaten - e.elocardinal * d.n) DESC
LIMIT 5;
Every row instead matches EloCardinal * (N - 1), where N is the number of
individuals ranked that day, within 0.00000006 across all three sources --
i.e. ExpNumBeaten compares each individual against every other ranked
individual, not against themselves.
Solution
Approved by the investigator on 2026-09-22: correct
doc/include/elo_ranks_daily_macro.m4 to describe ExpNumBeaten as
EloCardinal * (N - 1). Source ExpNumBeaten values are copied into
production unchanged regardless; only the written definition changes, and
this excludes no rows.
Problem
The shared elo_ranks_daily_indexes_create and
elo_ranks_daily_indexes_drop macros ignored their sex and community
arguments and hard-coded every index name and target relation to
ELO_RANKS_DAILY_KK_F. Consequently, KK_F had its documented unique and
secondary indexes, while KK_M, MT_F, and MT_M had only their primary-key
indexes. In particular, production did not enforce one row per
(Date, Individual) on those three tables.
Bad Data
SELECT tablename, indexname, indexdef FROM pg_indexes WHERE schemaname = 'sokwedb' AND tablename LIKE 'elo_ranks_daily_%' ORDER BY tablename, indexname;
Before the repair this returned nine indexes for KK_F and only the primary
key for each of KK_M, MT_F, and MT_M. A rollback-only probe confirmed that
KK_M accepted a duplicate (Date, Individual) row.
Solution
Implemented on 2026-09-23: both owning macros now derive the relation and index names from their sex and community arguments. All four generated create/drop pairs were reviewed independently and installed, leaving each table with its primary key plus the expected unique and seven secondary indexes. Rollback-only probes verified that duplicate INSERT and UPDATE collisions are rejected in KK_F, KK_M, MT_F, and MT_M, and that all probe rows are removed by rollback.
(#263) BIOGRAPHY_DATA reverse sex validation omits three ELO_RANKS_DAILY tables
Problem
Each ELO_RANKS_DAILY table's own trigger requires its Individual to have the
sex encoded by the table suffix. The reverse BIOGRAPHY_DATA trigger,
however, checked only ELO_RANKS_DAILY_KK_F. After a valid row was inserted,
changing the referenced biography sex could therefore leave KK_M, MT_F, or
MT_M in a state their own insert/update trigger would reject.
Bad Data
The defect was reproduced with rollback-only female and male biography fixtures. After inserting one valid rank row in turn, the following updates were accepted for MT_F, KK_M, and MT_M, while the equivalent KK_F update was rejected:
UPDATE sokwedb.biography_data SET sex = 'M' WHERE animid = 'female fixture'; UPDATE sokwedb.biography_data SET sex = 'F' WHERE animid = 'male fixture';
Solution
Implemented on 2026-09-23: the owning biography trigger M4 now uses one parameterized reverse-check fragment for KK_F, KK_M, MT_F, and MT_M. Focused rollback-only tests verified that female-to-male changes are rejected when the individual is referenced by KK_F or MT_F, male-to-female changes are rejected when referenced by KK_M or MT_M, and no fixture rows remain.
* (#264) BIOGRAPHY_DATA REPRO_STATES diagnostic can produce a NULL RAISE option
Problem
While selecting an existing female for the Problem #263 rollback probe,
changing AND from female to male reached the intended reverse REPRO_STATES
integrity check but failed while constructing its diagnostic. Several
REPRO_STATES fields included in DETAIL are nullable and are concatenated
without textualize(). A NULL field makes the entire DETAIL expression
NULL, which PL/pgSQL rejects before it can raise the intended
integrity_constraint_violation message.
Bad Data
BEGIN; UPDATE sokwedb.biography_data SET sex = 'M' WHERE animid = 'AND'; ROLLBACK;
This rollback-only probe fails with SQLSTATE 22004, RAISE statement option
cannot be null, in lib.biography_data_func() instead of reporting the
referencing REPRO_STATES row.
Solution
Unresolved and outside the Elo loader scope. Use textualize() or otherwise
NULL-safe formatting for every nullable REPRO_STATES value included in the
trigger diagnostic. The Elo reverse-trigger test uses isolated biography
fixtures so this pre-existing error path does not mask Problem #263.
* (#265) BIOGRAPHY_DATA ARRIVALS diagnostic references undeclared a_pid
Problem
While selecting an existing male for the Problem #263 rollback probe,
changing AL from male to female reached the ARRIVALS check for a female
assigned the male swelling code. The block's diagnostic references a_pid,
but the block neither declares that variable nor selects roles.pid into it.
PostgreSQL therefore fails while parsing the intended diagnostic expression.
Bad Data
BEGIN; UPDATE sokwedb.biography_data SET sex = 'F' WHERE animid = 'AL'; ROLLBACK;
This rollback-only probe fails with SQLSTATE 42703, column "a_pid" does not
exist, in lib.biography_data_func() instead of reporting the referencing
ARRIVALS row.
Solution
Unresolved and outside the Elo loader scope. Declare a_pid, select
roles.pid into it in each applicable ARRIVALS query, and retain it in the
diagnostic. The Elo reverse-trigger test uses isolated biography fixtures so
this pre-existing error path does not mask Problem #263.
* (#266) FERTILITY_FUNC's overlap check misses contained periods and skips revalidation on Study/AnimID changes
Problem
This is not a data conversion problem. It is a pre-existing defect in production trigger logic, discovered incidentally while working on FERTILITY-related conversion problems (see Problems #155 through #158). The conversion-issues log has no better venue for it, so it is recorded here rather than left undocumented.
lib.fertility_func(), defined in
db/schemas/lib/triggers/create/fertility.m4, enforces that, per
Study per AnimID, FERTILITY.StartDate/FERTILITY.StopDate
periods must not overlap. The enforcement has two independent gaps.
First, the overlap predicate only tests whether one of NEW's own
endpoints falls inside an existing row's interval:
SELECT fertility.id, fertility.study, fertility.animid
, fertility.startdate, fertility.starttype
, fertility.stopdate, fertility.stoptype
INTO a_id , a_study , a_animid
, a_startdate , a_starttype
, a_stopdate , a_stoptype
FROM fertility
WHERE fertility.id <> NEW.id
AND fertility.study = NEW.study
AND fertility.animid = NEW.animid
AND ((fertility.startdate <= NEW.startdate
AND fertility.stopdate >= NEW.startdate)
OR (fertility.startdate <= NEW.stopdate
AND fertility.stopdate >= NEW.stopdate))
-- Produce a consistent error message
ORDER BY fertility.id;
This misses the case where NEW's period fully contains an existing,
narrower period: neither of NEW.startdate/NEW.stopdate then falls
inside the existing row, so neither disjunct is true and no overlap is
reported. For example, an existing row with StartDate = 2000-03-01,
StopDate = 2000-03-31 does not conflict, by this test, with a new row
StartDate = 2000-01-01, StopDate = 2000-12-31, even though the second
period entirely contains the first.
Second, the trigger only re-runs this check on UPDATE when
StartDate or StopDate changes:
IF TG_OP = 'INSERT'
OR (NEW.startdate <> OLD.startdate
OR NEW.stopdate <> OLD.stopdate) THEN
-- Periods of fertility cannot overlap.
An UPDATE that changes Study or AnimID instead -- moving a
period to a different study or a different female -- skips the check
entirely, even though the row now belongs to a different (Study,
AnimID) partition that was never checked for overlap against the row's
new period. The comparisons are also NULL-unsafe: <> evaluates to
NULL, not TRUE, when either side is NULL, so a change into or
out of a NULL StartDate/StopDate would likewise fail to
trigger revalidation.
A related, cosmetic defect sits in the same code: the DETAIL error
message loses a closing parenthesis after StopType, so the message
reads ...Value (StopType) = (FOO: Overlapping row has... instead of
matching the ): Value (...) pattern used for every other field:
|| NEW.stoptype
|| ': Overlapping row has Key (FERTILITY.ID) = ('
doc/src/analyzed/fertility.m4 documents the same incomplete rule the
code implements, and would need to be corrected alongside any fix:
For any given female, for any given study, the periods of fertility cannot overlap. This means that, per female, per study, the |FERTILITY.StopDate| of any given row cannot be between, inclusive, the |FERTILITY.StartDate| and the |FERTILITY.StopDate| of any other row.
This prose has the same gap as the code: it defines overlap only in terms of
one row's StopDate falling within another row's range, so it does not
describe the containment case either.
Solution
Not addressed at this time. A candidate fix was prepared, verified against the standard inclusive-interval-overlap test, but never committed, and is recorded here for whoever picks this up next.
In db/schemas/lib/triggers/create/fertility.m4, replace the overlap
predicate with the standard two-sided inclusive-interval test, which covers
containment and shared endpoints in addition to the cases the original
predicate already caught:
AND fertility.startdate <= NEW.stopdate
AND NEW.startdate <= fertility.stopdate
Broaden the UPDATE revalidation guard to cover every column the
overlap check depends on, using IS DISTINCT FROM so a change into or
out of NULL is not silently ignored:
IF TG_OP = 'INSERT'
OR NEW.study IS DISTINCT FROM OLD.study
OR NEW.animid IS DISTINCT FROM OLD.animid
OR NEW.startdate IS DISTINCT FROM OLD.startdate
OR NEW.stopdate IS DISTINCT FROM OLD.stopdate THEN
-- Periods of fertility cannot overlap.
Restore the missing closing parenthesis in the DETAIL message so it
reads ): Overlapping row has..., matching every other field in the
same message.
In doc/src/analyzed/fertility.m4, correct the prose definition of the
overlap rule to state the actual (fixed) rule, including the containment and
shared-endpoint cases:
For any given female, for any given study, the periods of fertility cannot overlap, endpoints included. Two rows overlap when the |FERTILITY.StartDate| of each row is on or before the |FERTILITY.StopDate| of the other row. This includes periods that share an endpoint and periods in which one period contains the other.
Before committing, confirm the corrected predicate does not reject any period already present in the loaded production data; if it does, those rows are genuine, previously-undetected overlaps in the source and must be investigated on their own terms rather than folded into this fix.