sokwedb - User contributions [en]

Conversion Data Issues

2026-09-30T19:33:35Z

StevanEarl: Problem #266. Documents a code defect (not a conversion problem); added here since the code is static following the handoff to Ode

This page lists all the problems with the data that were encountered during the data conversion process, and how the issue was resolved.

The problems are numbered, in the order in which they were encountered during the conversion.

Given a choice, earlier problems should be solved before later problems.
This allows the later steps in the conversion to receive "correct" data, which help eliminate spurious problems, and aids the discovery of problems hidden by bad data.

Unsolved problems are marked with an <code>*</code>.

== (#1) FOLLOW_MAP_TIME duplicate keys ==

=== Problem ===
The data dump says that the <code>FOLLOW_MAP_TIME</code> table has a primary key consisting of, in order, the columns: <code>FMT_FOL_date</code>, <code>FMT_FOL_B_focal_AnimID</code>, <code>FMT_time</code>.
But these columns contain duplicate values.

=== Bad Data ===
The duplicate values can be listed (from the <code>raw</code> schema) with:

'''This query no longer reports results because the data was changed in the original MS Access data.'''

<pre>SELECT *
FROM "FOLLOW_MAP_TIME"
JOIN (SELECT "FMT_FOL_date" AS the_date
, "FMT_FOL_B_focal_AnimID" AS the_animid
, "FMT_time" AS the_time
FROM "FOLLOW_MAP_TIME"
GROUP BY "FMT_FOL_date", "FMT_FOL_B_focal_AnimID", "FMT_time"
HAVING count(*) > 1
) AS fmt
ON ("FOLLOW_MAP_TIME"."FMT_FOL_date" = fmt.the_date
AND "FOLLOW_MAP_TIME"."FMT_FOL_B_focal_AnimID" = fmt.the_animid
AND "FOLLOW_MAP_TIME"."FMT_time" = fmt.the_time);</pre>

=== Solution ===

Fixed in MS Access 11/17/23 by ICG. Checked all against Tikis.
When run in SokweDB, table does not exist

== (#2) SUBADULT_ARRIVALS_LOG has a textual SA_first_tiki_date column ==

=== Problem ===

The SUBADULT_ARRIVALS_LOG table has a column that is supposed to contain a date, but instead contains the string "TEXTY".

Likely, the entire row is bad. The row contains:

<pre>
SA_B_AnimID | SA_first_tiki_date | SA_notes
-------------+--------------------+----------
TEXTY | TEXTY | TEXTY</pre>

=== Bad Data ===

In the <code>raw</code> schema run:

'''This query no longer reports results because the data was changed in the original MS Access data.'''

<pre>
select * from "SUBADULT_ARRIVALS_LOG" where "SA_first_tiki_date" = 'TEXTY';</pre>

=== Solution ===

Fixed in MS Access, 11/16/2023, ICG.
Deleted row

== (#3) BRECORD_NOTES contains rows where BREC_FOL_date has values that are not just a date ==

=== Problem ===

There are 4 rows in BRECORD_NOTES where the BREC_FOL_date column, supposedly a date, contains time values that are not '00:00:00'.

=== Bad Data ===

In the <code>raw</code> schema run:

'''This query no longer reports results because the data was changed in the original MS Access data.'''

<pre>
select * from "BRECORD_NOTES" where "BREC_FOL_date"::TIME <> '00:00:00';</pre>

=== Solution ===

Fixed in MS Access, 11/16/2023, ICG
Deleted time stamp

== (#4) MATING_EVENT contains rows where M_FOL_date has values that are not just a date ==

=== Problem ===

The MATING_EVENT table contains 1 row where the time portion of M_FOL_date is not '00:00:00'.

=== Bad Data ===

In the <code>raw</code> schema run:

'''This query no longer reports results because the data was changed in the original MS Access data.'''

<pre>
select * from "MATING_EVENT" where "M_FOL_date"::TIME WITHOUT TIME ZONE <> '00:00:00';</pre>

=== Solution ===

Fixed in MS Access, 11/16/2023, ICG
Deleted time stamp

== (#5) BIOGRAPHY.DepartdateError data discarded ==

=== Problem ===

The BIOGRAPHY.DepartdateError column contains data that was, after discussion, determined to be unusable.

=== Bad Data ===

In the <code>raw</code> schema run:

'''This query no longer reports results because the data was changed in the original MS Access data.'''

<pre>
select * from raw."BIOGRAPHY" where "B_DepartdateError" <> 0;
</pre>

=== Solution ===
Do not convert the data. There is no corresponding column in the new db design.

== (#6) BIOGRAPHY.B_AnimID_num column contains the empty string ==

=== Problem ===
The BIOGRAPHY.B_AnimID_num column contains the empty string, instead of NULL.

=== Bad Data ===

In the <code>raw</code> schema run:

'''This query no longer reports results because the data was changed in the original MS Access data.'''

<pre>
select * from raw."BIOGRAPHY" where "B_AnimID_num" = '';
</pre>

=== Solution ===

Changed 11 empty string values to NULL in MS Access. ICG 12/6/2023

== (#7) BIOGRAPHY.B_AnimID_num column is textual ==

=== Problem ===
The BIOGRAPHY.B_AnimID_num column contains has a data type of TEXT.
The data values all begin with "CH".

=== Bad Data ===

In the <code>raw</code> schema run:

'''This query no longer reports results because the data was changed in the original MS Access data.'''

<pre>
select "B_AnimID", "B_AnimID_num"
from raw."BIOGRAPHY"
where "B_AnimID_num" <> ''
AND "B_AnimID_num" IS DISTINCT FROM NULL
order by "B_AnimID";
</pre>

=== Solution ===

Removed the "CH" prefix and made the column an integer in MS Access. ICG 12/6/2023

== (#8) BIOGRAPHY.DadID_publication_info column contains NULL values ==

No longer a problem as NULLs are now allowed.

=== (Not) Problem ===
The BIOGRAPHY.DadID_publication_info column has a data type that allows <code>NULL</code> values.
SokweDB wants only text

=== Bad Data ===

In the <code>raw</code> schema run:

'''This query no longer reports results because the data was changed in the original MS Access data.'''

<pre>
select *
from raw."BIOGRAPHY"
where "B_DadID" IS NULL
order by "B_AnimID";
</pre>

=== Solution ===

Changed the <code>NULL</code> to the empty string ("") in MS Access. ICG 12/6/2023

== (#9) BIOGRAPHY.DadID column contains non-AnimID values ==

=== Problem ===
The BIOGRAPHY.DadID column has a non-AnimID values.

=== Bad Data ===

In the <code>raw</code> schema run:
<pre>
select "B_AnimID", "B_DadID"
from raw."BIOGRAPHY"
where "B_DadID" is not null
and "B_DadID" <> ''
and not exists (select 1
from raw."BIOGRAPHY" as search
where search."B_AnimID" = raw."BIOGRAPHY"."B_DadID");
</pre>

=== Solution ===

Create the DadIDPrelim column in a BIOGRAPHY_DATA table, and make a BIOGRAPHY view that combines Dad_ID and DadIDPrelim into DadID -- adding the '_prelim' suffix as expected.

== (#10) BRECORD_NOTES contains rows where BREC_time has values that are not just a time ==

=== Problem ===

There are 56 rows in BRECORD_NOTES where the BREC_time column, supposedly a time, contains date values that are not '1899-12-30'.

=== Bad Data ===

In the <code>raw</code> schema run:
<pre>
select * from raw."BRECORD_NOTES" where "BREC_time"::DATE <> '1899-12-30';</pre>

=== Solution ===
The data is fixed in the conversion process.

== (#11) KAZ has b_dadid_publication_info, but b_dadid is NULL ==

=== Problem ===

<code>KAZ</code> has dad id publication info of 'Rudicell et al. 2010', but a <code>NULL</code> dadid.

=== Bad Data ===

In the <code>clean</code> schema run:
<pre>
select *
from clean.biography
where b_dadid is NULL
and (b_dadid_publication_info <> ''
or b_dadid_publication_info is null);</pre>

=== Solution ===

<code>KAZ</code> has 2 potential dads. Introduce a DadIDStatus column, to replace the DadIDPrelim column and have a code that describes what's going on with <code>KAZ</code>.

Fixed in the MS Access data; <code>KAZ</code> was assigned the <code>UNK</code> individual as the dad. Problem will be marked resolved with the upload of the next MS Access database dump.

== (#12) 9 BIOGRAPHY rows have BirthComm values that are not COMM_IDS.CommID values ==

=== Problem ===

There are 9 BIOGRAPHY rows with BirthComm values that are not valid COMM_IDS, their communities do not exist.

=== Bad Data ===

In the <code>clean</code> schema run:
<pre>
select *
from easy.biography
where b_birthgroup is not NULL
and not exists (select 1
from easy.community_lookup
where community_lookup.cl_community_id
= biography.b_birthgroup);</pre>

=== Solution ===
Added KL and KL_KK to CommIds. KL = Kalande, KL_KK= Kasekela/kalande
codes are created during conversion

Fixed in commit 5d85b83bfe1a.

== (#13) COMM_MEMBS rows place individuals in a community, that is not their birth community, before their BIOGRAPHY.EntryDate ==

=== Problem ===

13 COMM_MEMBS rows place individuals into a community, that is not their birth community, before BIOGRAPHY says they
entered the community.

''update: 12 rows as of 2026-07-19''

=== Bad Data ===

In the <code>clean</code> schema run:
<pre>
select b.b_animid, b.b_birthgroup, b.b_entrydate, cm.cm_start_date
from clean.community_membership as cm
join clean.biography as b
on (b.b_animid = cm.cm_b_animid)
where b.b_birthgroup is distinct from cm.cm_cl_community_id
and cm.cm_start_date < b.b_entrydate
order by b.b_animid, cm.cm_start_date;
</pre>

=== Solution ===
Talk to Karl about kk_P0 and kk_P1

== (#14) COMM_MEMBS rows place individuals in a community after their BIOGRAPHY.EndDate ==

=== Problem ===

3 COMM_MEMBS rows place individuals into a community after BIOGRAPHY says they
left the community.

=== Bad Data ===

In the <code>clean</code> schema run:
<pre>
select b.b_animid, cm.cm_end_date
from clean.community_membership as cm
join clean.biography as b
on (b.b_animid = cm.cm_b_animid)
where cm.cm_end_date > b.b_departdate
order by b.b_animid, cm.cm_end_date;
</pre>

=== Solution ===
ICG EVL fixed BH, HAI, TZB2 in Access. Need to talk to Karl about MG, RO, WD. Treat like KL chimps in Biography?

== (#15) TT is placed in a community twice on the same day ==

=== Problem ===

An individual may not be in more than one community (or even twice in the same community) on any given day.

=== Bad Data ===

In the <code>clean</code> schema run:
<pre>
select first.cm_b_animid as anim_id
, first.cm_start_date as first_start_date
, first.cm_end_date as first_end_date
, first.cm_cl_community_id as first_community_id
, first.cm_start_source as first_start_source
, first.cm_end_source as first_end_source
, second.cm_start_date as second_start_date
, second.cm_end_date as second_end_date
, second.cm_cl_community_id as second_community_id
, second.cm_start_source as second_start_source
, second.cm_end_source as second_end_source
from clean.community_membership as first
join clean.community_membership as second
on (first.cm_b_animid = second.cm_b_animid
and first.cm_start_date < second.cm_start_date)
where first.cm_end_date >= second.cm_start_date;
</pre>

=== Solution ===
Fixed in MS Access 2/1/2024
Changed end date of KK_P1 membership to 8/14/2022

ICG

== (#16) There are 31 rows in COMM_MEMB_LOG where MadeBy is NULL ==

=== Problem ===

There are 31 rows in COMM_MEMB_LOG where MadeBy is NULL, and the column does not allow NULL values.

=== Bad Data ===

In the <code>clean</code> schema run:
<pre>
select * from clean.community_membership_update_log where made_by is null;
</pre>

=== Solution ===

See problem #18. Fixed in commit 7a48fb2c4aad3b5bd4

== (#17) There are 64 rows in COMM_MEMB_LOG where the chimp_id is not on BIOGRAPHY ==

=== Problem ===

There are 64 rows in COMM_MEMB_LOG where the chimp_id is not a BIOGRAPHY.AnimID.

=== Bad Data ===

In the <code>clean</code> schema run:
<pre>
select date_of_update, chimp_id
from clean.community_membership_update_log as log
where not exists (select 1
from clean.biography
where biography.b_animid = log.chimp_id);
</pre>

=== Solution ===
Make this a "soft" error. Fixed in commit 4c9626b304.

== (#18) There are 31 rows in COMM_MEMB_LOG where the MadeBy is NULL ==

=== Problem ===

There are 31 rows in COMM_MEMB_LOG where MadeBy is NULL, and the column does not allow
NULLs.

=== Bad Data ===

In the <code>clean</code> schema run:
<pre>
select * from clean.community_membership_update_log where made_by is null;
</pre>

=== Solution ===

Create an unknown person (UNK), and when the person is NULL, use the unkonwn person.

This allows us to make the unknown person "inactive", preventing them from being used
in newly entered data. The alternative, allowing NULL MadeBy values, would allow
new "bad data", and NULL values make querying harder.

Fixed in commit 7a48fb2c4aad3b5b.

== (#19) There are 18 rows in BIOGRAPHY_LOG where the MadeBy is NULL ==

=== Problem ===

There are 18 rows in BIOGRAPHY_LOG where MadeBy is NULL, and the column does not allow
NULLs.

=== Bad Data ===

In the <code>clean</code> schema run:
<pre>
select * from clean.biography_update_log where made_by is null;
</pre>

=== Solution ===

Create an unknown person (UNK), and when the person is NULL, use the unkonwn person.
(See Problem #18)== * (#19) There are 18 rows in BIOGRAPHY_LOG where the MadeBy is NULL ==

Fixed in commit 8a528cc7d1a9fb.

''update 2026-07-26:''

The 18 problematic values are zero-length strings, not NULL, so they pass through the ELSE made_by branch and violate the foreign key as madeby=(). This solution was hardened with commit 6279986f5f77cab69d19733925f173eadca23b60.

== (#20) There are 3 rows in BIOGRAPHY_LOG where the Rationale is NULL ==

=== Problem ===

There are 3 rows in BIOGRAPHY_LOG where Rationale is NULL. The column does not allow
NULLs; normally the conversion program would convert NULL to the empty string.
But the Rationale column requires there be (non-empty) textual data.

=== Bad Data ===

In the <code>clean</code> schema run:
<pre>
select * from clean.biography_update_log where update_rationale is null;
</pre>

=== Solution ===
ICG fixed in MS Access 2/8/2024. Updated rationale to 'routine update'.

== (#21) There are 6 rows in BIOGRAPHY_LOG where the update_escription is NULL ==

=== Problem ===

There are 6 rows in BIOGRAPHY_LOG where Description is NULL.
It appears that the description was put into the Rationale column, sometimes along with some rationale.

=== Bad Data ===

In the <code>clean</code> schema run:
<pre>
select * from clean.biography_update_log where update_description is null;
</pre>

=== Solution ===
ICG fixed in MS Access 2/8/2024, using information in update_rationale

== (#22) There are 85 rows in BIOGRAPHY_LOG where the chimp_id is not on BIOGRAPHY ==

=== Problem ===

There are 85 rows in BIOGRAPHY_LOG where the chimp_id is not a BIOGRAPHY.AnimID.

=== Bad Data ===

In the <code>clean</code> schema run:
<pre>
select *
from clean.biography_update_log as log
where not exists (select 1
from clean.biography
where biography.b_animid = log.chimp_id);
</pre>

=== Solution ===
Make this a "soft" error. Fixed in commit c96555f9f326.

== (#23) There are 65 rows in BIOGRAPHY_LOG where the MadeBy is not on PEOPLE ==

=== Problem ===

There are 65 rows in BIOGRAPHY_LOG where the MadeBy is not on PEOPLE, but there is MadeBy data.

These are "combination" ID errors, where multiple people are entered instead
of a single people code.

=== Bad Data ===

In the <code>clean</code> schema run:
<pre>
select *
from clean.biography_update_log as log
where not exists (select 1
from clean.people

</pre>
where people.person = log.made_by)
and made_by is not null;
</pre>

=== Solution ===
Fixed in MS Access by ICG 2/8/2024. Changed multiple IDs to the one who made the final change.

== (#24) Follow starts are not first arrivals ==

=== Problem ===
There are 6,397 cases where the old FOLLOW table's columns
"fol_time_begin" is not the first arrival time, fa_time_start,
on FOLLOW_ARRIVAL.

The The Gombe Chimpanzee Database Handbook says that this
should never happen, because the follow start is supposed
to be the first arrival. Is there additional data
in the old fol_time_begin, like the actual start
the observers started working? If not, which data is correct,
the one in FOLLOW or the one in FOLLOW_ARRIVAL?

=== Bad Data ===
<pre>
WITH spans AS
(SELECT fa_fol_date, fa_fol_b_focal_animid
, MIN(fa_time_start) AS min_start
FROM clean.follow_arrival
WHERE fa_b_arr_animid = fa_fol_b_focal_animid
GROUP BY fa_fol_date, fa_fol_b_focal_animid)
SELECT spans.*, follow.fol_time_begin
FROM spans
JOIN clean.follow
ON (follow.fol_date = spans.fa_fol_date
AND follow.fol_b_animid = spans.fa_fol_b_focal_animid)
WHERE NOT EXISTS
(SELECT 1
FROM clean.follow
WHERE follow.fol_date = spans.fa_fol_date
AND follow.fol_b_animid = spans.fa_fol_b_focal_animid
AND spans.min_start = follow.fol_time_begin)
ORDER BY spans.fa_fol_date, spans.fa_fol_b_focal_animid;
</pre>

=== Solution ===

This is data entry error, ignore the problem and ignore the data in the follow.fol_time_begin.

==(#25) Follow ends are not last arrivals ==

=== Problem ===
There are 6,570 cases where the old FOLLOW table's columns
"fol_time_end" is not the last arrival
time, fa_time_end, on FOLLOW_ARRIVAL.

The The Gombe Chimpanzee Database Handbook says that this
should never happen, because the follow end is supposed
to be the first arrival/last departure. Is there additional data
in the old fol_time_end, like the actual end-time
the observers started working? If not, which data is correct,
the one in FOLLOW or the one in FOLLOW_ARRIVAL?

=== Bad Data ===
<pre>
WITH spans AS
(SELECT fa_fol_date, fa_fol_b_focal_animid
, MAX(fa_time_end) AS max_end
FROM clean.follow_arrival
WHERE fa_b_arr_animid = fa_fol_b_focal_animid
GROUP BY fa_fol_date, fa_fol_b_focal_animid)
SELECT spans.*, follow.fol_time_end
FROM spans
JOIN clean.follow
ON (follow.fol_date = spans.fa_fol_date
AND follow.fol_b_animid = spans.fa_fol_b_focal_animid)
WHERE NOT EXISTS
(SELECT 1
FROM clean.follow
WHERE follow.fol_date = spans.fa_fol_date
AND follow.fol_b_animid = spans.fa_fol_b_focal_animid
AND spans.max_end = follow.fol_time_end)
ORDER BY spans.fa_fol_date, spans.fa_fol_b_focal_animid;
</pre>

=== Solution ===

This is data entry error, ignore the problem and ignore the data in the follow.fol_time_end.

== (#26) Mismatch of start-in-nest on follow and follow_arrival ==

=== Problem ===

There are <s>1,848</s> 3,369 follows where the follow arrival says the focal started in the nest but the follow does not, or vice-versa.

=== Bad Data ===

<pre>
WITH spans AS
(SELECT fa_fol_date, fa_fol_b_focal_animid
, MIN(fa_time_start) AS min_start
FROM easy.follow_arrival
WHERE fa_b_arr_animid = fa_fol_b_focal_animid
GROUP BY fa_fol_date, fa_fol_b_focal_animid)
SELECT spans.*, follow.fol_time_begin, follow.fol_flag_begin_in_nest
, first_arrivals.fa_type_of_nesting
FROM easy.follow
JOIN spans
ON (follow.fol_date = spans.fa_fol_date
AND follow.fol_b_animid = spans.fa_fol_b_focal_animid)
JOIN easy.follow_arrival
AS first_arrivals
ON (first_arrivals.fa_fol_date = follow.fol_date
AND first_arrivals.fa_fol_b_focal_animid = follow.fol_b_animid
AND first_arrivals.fa_time_start = spans.min_start)
WHERE (((first_arrivals.fa_type_of_nesting = 1
OR first_arrivals.fa_type_of_nesting = 3)
AND follow.fol_flag_begin_in_nest = 0)
OR (first_arrivals.fa_type_of_nesting <> 1
AND first_arrivals.fa_type_of_nesting <> 3
AND follow.fol_flag_begin_in_nest = 1))
ORDER BY spans.fa_fol_date, spans.fa_fol_b_focal_animid;
</pre>

=== Solution ===

Ignore. Default to data from Follow_Arrival

==== Remarks ====

When applying the solution, solving both problem #26 and #27,
and updating follow_arrival so that it is the one source of truth
used by the conversion,
there are 297 follow_arrival rows updated.

This implies that it is primarily the follow table
that does not mark the individual as being in a
nest when they should be in a nest. ("Should be",
according to the accepted solution.)

==(#27) Mismatch of end-in-nest on follow and follow_arrival ==

=== Problem ===

There are 3,804 follows where the follow arrival says the focal ended in the nest but the follow does not, or vice-versa.

=== Bad Data ===

<pre>
WITH spans AS
(SELECT fa_fol_date, fa_fol_b_focal_animid
, MAX(fa_time_end) AS max_end
FROM easy.follow_arrival
WHERE fa_b_arr_animid = fa_fol_b_focal_animid
GROUP BY fa_fol_date, fa_fol_b_focal_animid)
SELECT spans.*
, follow.fol_time_end, follow.fol_flag_end_in_nest
, last_arrivals.fa_type_of_nesting
FROM easy.follow
JOIN spans
ON (follow.fol_date = spans.fa_fol_date
AND follow.fol_b_animid = spans.fa_fol_b_focal_animid)
JOIN easy.follow_arrival
AS last_arrivals
ON (last_arrivals.fa_fol_date = follow.fol_date
AND last_arrivals.fa_fol_b_focal_animid = follow.fol_b_animid
AND last_arrivals.fa_time_end = spans.max_end)
WHERE (((last_arrivals.fa_type_of_nesting = 2
OR last_arrivals.fa_type_of_nesting = 3)
AND follow.fol_flag_end_in_nest = 0)
OR (last_arrivals.fa_type_of_nesting <> 2
AND last_arrivals.fa_type_of_nesting <> 3
AND follow.fol_flag_end_in_nest = 1))
ORDER BY spans.fa_fol_date, spans.fa_fol_b_focal_animid;
</pre>

=== Solution ===

Ignore the problem. Default to Follow_Arrival

==== Remarks ====

See the remarks for [[Conversion Data Issues#(#27) Mismatch of end-in-nest on follow and follow_arrival|problem #26]].

== (#28) The FOLLOW.FOL_distance_traveled column is not converted ==

=== Problem ===

The FOLLOW.FOL_distance_traveled column has no corresponding column in the new
database design. The conversion process does not check that the value
of this column is consistent with the other data in the database from which it is computed.

=== Solution ===

This is a computed column and does not need to be converted. The desired value is computed in the design of the new database.

The assumption is that the "raw" data from which this value is computed in the MS Access database is correct.

== (#29) The FOLLOW.Brecord_notes column is not converted ==

=== Problem ===

The FOLLOW.Brecord_notes column has no corresponding column in the new
database design.

=== Solution ===

This column is used for administrative purposes and does not need to be in the new database design.

== (#30) Some follows have no community ==

=== Problem ===

There are 13 follows with no community.

=== Bad Data ===

<pre>
select *
from clean.follow
where fol_cl_community_id is null
order by fol_date, fol_b_animid;
</pre>

=== solution ===

Solved

== (#31) Some follows have an animid with trailing spaces ==

=== Problem ===

There are <s>9</s> 39 follows with animids that don't exist, because they have trailing spaces. These are comprised of <s>9</s> 23 distinct animids.

=== Bad Data ===

Cleaned up in <code>clean</code> schema, so query <code>easy</code> schema.

<pre>
select rtrim(fol_b_animid)
from easy.follow
where rtrim(fol_b_animid) <> fol_b_animid
order by fol_b_animid;
</pre>

=== Solution ===

Remove the trailing spaces in the conversion process.

== (#32) Some follows have an animid that is lower-case ==

=== Problem ===

There is 1 follows with an animids that don't exist, because it is lower-case.

=== Bad Data ===

Cleaned up in <code>clean</code> schema, so query <code>easy</code> schema.
<pre>
select *
from easy.follow
where upper(fol_b_animid) <> fol_b_animid
order by fol_b_animid;
</pre>

=== Solution ===

Convert the animid to upper-case in the conversion process.

==(#33) Some follows have an animid that does not exist, even after animid cleanup ==

=== Problem ===

There are 14 follows with animids that don't exist, even after cleanup that removes trailing spaces and forces upper-case. (Some of these may be due to prior errors.) These are comprised of 3 distinct animids.

=== Bad Data ===

<pre>
-- The 14 follows with bad animids
select *
from clean.follow
where not exists
(select 1
from clean.biography
where biography.b_animid = upper(rtrim(follow.fol_b_animid)))
order by fol_date, fol_b_animid;
</pre>

<pre>
-- The 3 animids involved
select distinct follow.fol_b_animid
from clean.follow
where not exists
(select 1
from clean.biography
where biography.b_animid = upper(rtrim(follow.fol_b_animid)))
order by follow.fol_b_animid;
</pre>

=== Solution ===
Solved

== (#34) There are duplicate animid, date combinations on FOLLOW ==

=== Problem ===
There are 2 sets of duplicate animid, date combinations on the FOLLOW table, for a total of 4 rows.

=== Bad Data ===

<pre>
-- The duplicate animid, date combinations
select follow.fol_b_animid, follow.fol_date, count(*)
from clean.follow
group by follow.fol_b_animid, follow.fol_date
having count(*) > 1
order by fol_b_animid, fol_date;
</pre>

<pre>
-- The duplicate rows
with dups as (
select follow.fol_b_animid, follow.fol_date
from clean.follow
group by follow.fol_b_animid, follow.fol_date
having count(*) > 1)
select *
from clean.follow
join dups on (follow.fol_b_animid = dups.fol_b_animid
and follow.fol_date = dups.fol_date)
order by follow.fol_b_animid, follow.fol_date;
</pre>
=== Solution ===
Solved

== (#35) There are follows done before a focal was under study ==

=== Problem ===
There are 5 follows that are done before their focal was under study, before the focal's EntryDate.

=== Bad Data ===

<pre>
select follow.*, biography.b_entrydate
from clean.follow
join clean.biography on (biography.b_animid = follow.fol_b_animid)
where biography.b_entrydate > follow.fol_date
order by follow.fol_date, follow.fol_b_animid;
</pre>

=== Solution ===

Solved

== (#36) FOLLOW_OBSERVERS.Period is not checked against follow start or stop times ==

=== Problem ===
The conversion process uses the clean.follow columns of fol_am_observer_1, fol_am_observer_2, fol_pm_observer_1, and fol_pm_observer_2 to populate FOLLOW_OBSERVERS.
(The *_observer_1 column going into FOLLOW_OBSERVERS.OBS_BRec and the *_observer_2 column going in OBS_Tiki.)

If both the *_observer_1 and the *_observer_2 columns are either NULL or the empty string (after space trimming), the no row is created for the respective time period.

There are no checks done to ensure that the time periods of the follow have any relation to the FOLLOW_OBSERVERS.Period value.

=== Solution ===

Ignore the problem. (If someone cares, add a query to the warning system.)

== (#37) Some follows have no recorded observers ==

=== Problem ===

There are 56 follows with no recorded observers, but the system requires there be a FOLLOW_OBSERVERS record related to the follow.

See problem #36 for a description of the follow observer conversion process.

=== Bad data ===

<pre>
select *
from clean.follow
where coalesce(btrim(fol_am_observer_1), '') = ''
and coalesce(btrim(fol_am_observer_2), '') = ''
and coalesce(btrim(fol_pm_observer_1), '') = ''
and coalesce(btrim(fol_pm_observer_2), '') = ''
order by fol_date, fol_b_animid;
</pre>

=== Solution ===

Make the "NONE" (no observer) person the observers.

Create a "UNK" (unknown) time period in PERIODS, and make that the time period.

== (#38) Some follows have only one observer, but the system wants two ==

=== Problem ===

There are follows with only one observer, but the system requires a FOLLOW_OBSERVERS record have a value in both OBS_BRec and OBS_Tiki.

See problem #36 for a description of the follow observer conversion process.

=== Solution ===

Create a "NONE" person, and make that person the observer when there's otherwise not a value.

== (#39) In follow, there are observers that are 2 people ==

=== Problem ===

In the follow table, in columns fol_am_observer_1, fol_am_observer_2, fol_pm_observer_1, fol_pm_observer_2, there are 30 rows (28, really, because of n/a) that appear to represent 2 people.

These look like names, separated by the "/" character.

=== Bad data ===
<pre>
WITH new_people AS (
SELECT BTRIM(follow.fol_am_observer_1) AS person
FROM clean.follow
WHERE follow.fol_am_observer_1 IS NOT NULL
GROUP BY BTRIM(follow.fol_am_observer_1)
UNION
SELECT BTRIM(follow.fol_am_observer_2) AS person
FROM clean.follow
WHERE follow.fol_am_observer_2 IS NOT NULL
GROUP BY BTRIM(follow.fol_am_observer_2)
UNION
SELECT BTRIM(follow.fol_pm_observer_1) AS person
FROM clean.follow
WHERE follow.fol_pm_observer_1 IS NOT NULL
GROUP BY BTRIM(follow.fol_pm_observer_1)
UNION
SELECT BTRIM(follow.fol_pm_observer_2) AS person
FROM clean.follow
WHERE follow.fol_pm_observer_2 IS NOT NULL
GROUP BY BTRIM(follow.fol_pm_observer_2)
)
, uniq_people AS (
SELECT new_people.person
FROM new_people
WHERE new_people.person <> ''
GROUP BY new_people.person
)
SELECT uniq_people.person
FROM uniq_people
JOIN uniq_people AS up
ON (LOWER(uniq_people.person) = LOWER(up.person))
WHERE uniq_people.person <> up.person
AND STRPOS(uniq_people.person, '/') <> 0
GROUP BY uniq_people.person
ORDER BY LOWER(uniq_people.person), uniq_people.person;
</pre>

=== Solution ===

Ignore the problem. Mark all people containing a "/" character as not active, so they can't be used in the future.

== (#40) In follow, there are observers that differ only by character case ==

=== Problem ===

In the follow table, in columns fol_am_observer_1, fol_am_observer_2, fol_pm_observer_1, fol_pm_observer_2, there are 511 names that differ only by character case. So, about half that in terms of unique names.

This is complicated to account for and exclude duplicates in the conversion process. So resolution of this is holding back additional conversion work.

=== Bad data ===

Note the use of the <code>easy</code> schema, instead of the
<code>clean</code> schema. This is because the observer data
in the <code>clean</code> schema has been case-normalized.

<pre>
WITH new_people AS (
SELECT BTRIM(follow.fol_am_observer_1) AS person
FROM easy.follow
WHERE follow.fol_am_observer_1 IS NOT NULL
GROUP BY BTRIM(follow.fol_am_observer_1)
UNION
SELECT BTRIM(follow.fol_am_observer_2) AS person
FROM easy.follow
WHERE follow.fol_am_observer_2 IS NOT NULL
GROUP BY BTRIM(follow.fol_am_observer_2)
UNION
SELECT BTRIM(follow.fol_pm_observer_1) AS person
FROM easy.follow
WHERE follow.fol_pm_observer_1 IS NOT NULL
GROUP BY BTRIM(follow.fol_pm_observer_1)
UNION
SELECT BTRIM(follow.fol_pm_observer_2) AS person
FROM easy.follow
WHERE follow.fol_pm_observer_2 IS NOT NULL
GROUP BY BTRIM(follow.fol_pm_observer_2)
)
, uniq_people AS (
SELECT new_people.person
FROM new_people
WHERE new_people.person <> ''
GROUP BY new_people.person
)
SELECT uniq_people.person
FROM uniq_people
JOIN uniq_people AS up
ON (LOWER(uniq_people.person) = LOWER(up.person))
WHERE uniq_people.person <> up.person
GROUP BY uniq_people.person
ORDER BY LOWER(uniq_people.person), uniq_people.person;
</pre>

=== Solution ===

Use the mixed case code when such exists. This involves changing the observer columns in the <code>follow</code> table. See [[The_Old_Database#The_observer_columns_of_the_follow_table|the notes]].

==(#41)There are follow arrivals with NULL nesting information ==

=== Problem ===

In the follow_arrival table, there are 11 rows with a NULL fa_type_of_nesting.

=== Bad data ===

<pre>
select * from clean.follow_arrival where fa_type_of_nesting IS NULL;
</pre>

=== Solution ===

Solved

== (#42) There are follow arrivals with NULL sexual cycle information ==

=== Problem ===

In the follow_arrival table, there are 222 rows with a NULL fa_type_of_cycle.

=== Bad data ===

<pre>
select * from clean.follow_arrival where fa_type_of_cycle is null;
</pre>

=== Solution ===

Make a <code>MISS</code> <code>CYCLE_STATES</code> value and use that for the arrivals with no cycle information.

== * (#43) There are follow arrivals with no related follow ==

=== Problem ===

In the follow_arrival table, there are <s>732</s> <s>276</s> 75 rows with a focal and a date that have no matching information on the follow table.

=== Bad data ===

<pre>
SELECT *
FROM clean.follow_arrival
WHERE NOT EXISTS
(SELECT 1
FROM clean.follow
WHERE follow.fol_b_animid = follow_arrival.fa_fol_b_focal_animid
AND follow.fol_date = follow_arrival.fa_fol_date)
ORDER BY follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid;
</pre>
=== Solution ===
5/5/2026 ICG
FIXED MANY ROWS IN MS ACCESS

Most of the rest are hand-entered juvenile arrivals from Brec. Spot checks have Brec notes but no physical tiki
Need to investigate further - brec swahili? Eventually the rest of the individuals in the Brec notes need to be entered by hand.

Should be solved by the WATCHES table

For now, allow these rows

==== Remarks on Solution ====

There is no follow to attach the arrival to. No WATCHES row.
So there is no way to get the data in and simply write
a warning.

Allowing these rows requires a re-design of the database
and additional "hard" rules. Which is going to take time
to program, and delay finishing. And, it will take additional time
to rip-out the rules when you finally resolve the data issue.

What is needed is to make a new WATCHES.Type just to allow for arrivals
with no related follow, and ensure that only arrivals without
follows are attaching to this new Type.

The only "easy" alternative I can think of is:

Don't convert this data at this time. Convert it when
we get around to fixing the data. In the mean time,
anyone who wants to work with this data can get it
from the "clean" schema, using the query provided in
the problem #43 section, and figure out how to integrate
it with whatever they are working on.

==(#44) There are follow arrivals where the arriving chimp arrives before being under study ==

=== Problem ===

In the follow_arrival table, there are <s>217</s> 49 rows where the fa_b_arr_animid, the arriving individual, has an entry date after the date of the follow.

=== Bad data ===

<pre>
select follow_arrival.*, biography.b_entrydate, biography.b_sex
from clean.follow_arrival
join clean.biography
on (biography.b_animid = follow_arrival.fa_b_arr_animid)
where biography.b_entrydate > follow_arrival.fa_fol_date
order by biography.b_animid, follow_arrival.fa_fol_date;
</pre>
=== Solution ===
Something is wrong with community entry date or biography

SN1 fixed in MS Access ICG 5/19/2026

Elo Confirmed CT and SG were with CA on 10/23/1980 from brec swahili

RESOLVED

== (#45) The follow_arrival.fa_update column is not preserved ==

=== Problem ===

There is nowhere in the current design to store the values in the follow_arrival.fa_update column.

=== Solution ===

Add a temporal extension to Postgres to make the db into a temporal database to track change history and be able to see data as it existed at any point in time.

FOLLOWUP WITH KARL.
Will re-review this should a temporal extension be out of budget, etc.

ASK KARL WHY THIS COLUMN CAN'T GO IN

There is now a ARRIVALS.Updated column. This provides a place to put the existing
data. Going forward, a PG extension (see below) can be installed for more functionality.

Wouldn't you rather have a [https://en.wikipedia.org/wiki/Temporal_database temporal database]? (Simple description in the top paragraph [https://wiki.postgresql.org/wiki/Temporal_Extensions here].) This would allow you to "time travel" and look at the database content at any point in time. And, it puts a "last changed" timestamp on every row, of every table you say to track temporally, automatically.

If so, install one of [https://wiki.postgresql.org/wiki/Temporal_Extensions these extensions] to PostgreSQL. (Or, as a separate project, I will improve one of them so that the people who don't need to time travel don't see any changes to the existing database at all. What I don't like about most of the extensions listed is that they add extra columns to existing tables, as well as adding extra "history" tables in places where they clutter up what users really need to see.)

== (#46) There are follow_arrivals where non-females have a cycle code that is other than <code>n/a</code> ==

=== Problem ===

There are <s>169</s> 168 follow_arrival rows, for non-female arriving individuals, that have a sexual cycle code (fa_type_of_cycle) that is not <code>n/a</code>.

=== Bad data ===

<pre>
select follow_arrival.*, biography.b_sex
from clean.follow_arrival
join clean.biography
on (biography.b_animid = follow_arrival.fa_b_arr_animid)
where biography.b_sex <> 'F'
and follow_arrival.fa_type_of_cycle <> 'n/a'
order by follow_arrival.fa_fol_date
, follow_arrival.fa_b_arr_animid;
</pre>

To summarize by sex and cycle code:

<pre>
select biography.b_sex, follow_arrival.fa_type_of_cycle, count(*)
from clean.follow_arrival
join clean.biography
on (biography.b_animid = follow_arrival.fa_b_arr_animid)
where biography.b_sex <> 'F'
and follow_arrival.fa_type_of_cycle <> 'n/a'
group by biography.b_sex, follow_arrival.fa_type_of_cycle order by biography.b_sex
, follow_arrival.fa_type_of_cycle;
</pre>

=== Solution ===

FIX THE FEW ACTUAL MALES THAT DON'T HAVE AN n/a

Change non-females (INCLUDING UNKNOWN SEX) who have a cycle code of <code>0</code> to a cycle code of <code>n/a</code>. IAN TO FIX IN ACCESS

IAN FIXED IN ACCESS 3/25/2026

== * (#47) There are follow_arrivals where females that are too young have a cycle code of <code>U</code> ==

=== Problem ===

There are 16 follow_arrival rows, for female arriving individuals, that have a sexual cycle code (fa_type_of_cycle) of <code>U</code> but are less than 5 years of age.

Naturally, this number will change if the age limit is changed, but this is here as a placeholder.

Note: The test is against the birthdate, not the minimum possible birthdate.

=== Bad data ===

<pre>
SELECT fa.*
, b.b_sex
, b.b_birthdate
, AGE(fa.fa_fol_date, b.b_birthdate) AS age_at_follow
, INTERVAL '5 years'
- AGE(fa.fa_fol_date, b.b_birthdate) AS younger_than_five_by
FROM clean.follow_arrival AS fa
JOIN clean.biography AS b
ON b.b_animid = fa.fa_b_arr_animid
WHERE b.b_sex = 'F'
AND fa.fa_type_of_cycle = 'U'
AND b.b_birthdate > fa.fa_fol_date - INTERVAL '5 years'
ORDER BY fa.fa_fol_date
, fa.fa_fol_b_focal_animid
, fa.fa_b_arr_animid;
</pre>

Note: The original query used the wrong inequality symbol.

=== Solution ===

Changing the limit from 6 years of age to 5 reduced the number of outstanding problems to 16.

The rest will have to be fixed in the data. Alternately, we can adjust the hard limit, and set a soft limit of 5 years in the warning system with a note to change the hard limit back once the errors are resolved.

Resolution: Fixed in the data.

UPDATE 2026-07-23 it does not appear that these data were fixed, please see below for a summary of this issue and related Problem #49:

UPDATE 2026-08-14: fixed in Access by ICG

UPDATE 2026-08-19: improved but there are still 12 offending records

{| class="wikitable sortable"
|-
! problem !! problem_description !! age_years !! current_cycle !! youngest_exact_age !! oldest_exact_age !! offending_rows !! problem_total
|-
| 47 || U assigned before age 5 || 1 || U || 1 year 4 mons 5 days || 1 year 4 mons 5 days || 1 || 21
|-
| 47 || U assigned before age 5 || 2 || U || 2 years 11 mons 26 days || 2 years 11 mons 26 days || 1 || 21
|-
| 47 || U assigned before age 5 || 3 || U || 3 years 3 mons 30 days || 3 years 11 mons 1 day || 10 || 21
|-
| 47 || U assigned before age 5 || 4 || U || 4 years 5 mons 21 days || 4 years 9 mons 18 days || 9 || 21
|-
| 49 || U assigned at age 15 or older || 15 || U || 15 years 1 mon 2 days || 15 years 10 mons 14 days || 16 || 38
|-
| 49 || U assigned at age 15 or older || 16 || U || 16 years 19 days || 16 years 11 mons 6 days || 7 || 38
|-
| 49 || U assigned at age 15 or older || 17 || U || 17 years 1 mon 19 days || 17 years 1 mon 19 days || 1 || 38
|-
| 49 || U assigned at age 15 or older || 18 || U || 18 years 1 mon 18 days || 18 years 8 mons 16 days || 6 || 38
|-
| 49 || U assigned at age 15 or older || 32 || U || 32 years 1 mon 21 days || 32 years 5 mons 14 days || 4 || 38
|-
| 49 || U assigned at age 15 or older || 33 || U || 33 years 6 days || 33 years 6 days || 4 || 38
|}

option matrix:<br>
{| class="wikitable"
|-
! Value !! Meaning !! Accepted under age 5?
|-
| <code>0</code> || Not swollen || Yes
|-
| <code>MISS</code> || Missing data || Yes
|-
| <code>U</code> || Adolescent swelling || No
|-
| <code>0.25</code>, <code>0.5</code>, <code>0.75</code>, <code>1</code> || Increasing observed swelling || No—female must be at least 8
|-
| <code>n/a</code> || Male/not applicable || No—prohibited for females
|}

== * (#48) There are follow arrivals where the arriving chimp arrives after finishing being under study ==

=== Problem ===

In the follow_arrival table, there are <s>239</s> 324 rows where the fa_b_arr_animid, the arriving individual, has a departure date before the date of the follow.

=== Bad data ===

<pre>
select follow_arrival.*, biography.b_departdate, biography.b_sex
from clean.follow_arrival
join clean.biography
on (biography.b_animid = follow_arrival.fa_b_arr_animid)
where biography.b_departdate < follow_arrival.fa_fol_date
order by biography.b_animid, follow_arrival.fa_fol_date;
</pre>

=== SOLUTION ===
IAN TO FIX IN ACCESS - CHECK DATES OF FOLLOWS AND ACCURACY OF IDS

== * (#49) There are follow_arrivals where females that are too old have a cycle code of <code>U</code> ==

=== Problem ===

There are <s>127</s> <s>98</s> <s>86</s> 48 follow_arrival rows, for female arriving individuals, that have a sexual cycle code (fa_type_of_cycle) of <code>U</code> but are more than 14 years of age.
(Actually, because the endpoint takes up the whole 14th year, this means at least 15 years of age.)

Naturally, this number will change if the age limit is changed, but this is here as a placeholder.

Note: The test is against the birthdate, not the maximum possible birthdate.

=== Bad data ===

Note: Original query used the wrong inequality.

<pre>
SELECT fa.*
, b.b_sex
, b.b_birthdate
, AGE(fa.fa_fol_date, b.b_birthdate) AS age_at_follow
, AGE(fa.fa_fol_date, b.b_birthdate)
- INTERVAL '15 years' AS older_than_limit_by
FROM clean.follow_arrival AS fa
JOIN clean.biography AS b
ON b.b_animid = fa.fa_b_arr_animid
WHERE b.b_sex = 'F'
AND fa.fa_type_of_cycle = 'U'
AND b.b_birthdate
<= fa.fa_fol_date
- INTERVAL '1 year'
- INTERVAL '14 years'
ORDER BY fa.fa_fol_date
, fa.fa_fol_b_focal_animid
, fa.fa_b_arr_animid;
</pre>

=== Solution ===

The number of problem rows was reduced after changing the limit to 14 years from 9 years.

The remaining problems will have to be adjusted in the data. Alternately, we can adjust the hard limit, and set a soft limit of 14 years in the warning system with a note to change the hard limit back once the errors are resolved.

5/7/2026: ICG checked through 2003. When there was a "?" for swelling, I entered '0'. If there was actually a 'U' on the tiki itself, I left "U" in the FA table. Note that a lot of these females are MGF, so the U might be legit.

5/19/2026: MGFs have a dummy birthdate which is why the age is being flagged. The rest are what the observer recorded, so let them in.

UPDATE 2026-07-23: There are 86 failing rows; many but not all are MGF (how to we treat MGF?). Please see below for a summary of this issue and related Problem #47:<br>

8/14/2026 - ICG fixed in Access - all non-MGF entries changed to "0"

UPDATE 2026-08-14: Improved but there are still 48 offending records

''note that this summary does not include MFG''
{| class="wikitable sortable"
|-
! problem !! problem_description !! age_years !! current_cycle !! youngest_exact_age !! oldest_exact_age !! offending_rows !! problem_total
|-
| 47 || U assigned before age 5 || 1 || U || 1 year 4 mons 5 days || 1 year 4 mons 5 days || 1 || 21
|-
| 47 || U assigned before age 5 || 2 || U || 2 years 11 mons 26 days || 2 years 11 mons 26 days || 1 || 21
|-
| 47 || U assigned before age 5 || 3 || U || 3 years 3 mons 30 days || 3 years 11 mons 1 day || 10 || 21
|-
| 47 || U assigned before age 5 || 4 || U || 4 years 5 mons 21 days || 4 years 9 mons 18 days || 9 || 21
|-
| 49 || U assigned at age 15 or older || 15 || U || 15 years 1 mon 2 days || 15 years 10 mons 14 days || 16 || 38
|-
| 49 || U assigned at age 15 or older || 16 || U || 16 years 19 days || 16 years 11 mons 6 days || 7 || 38
|-
| 49 || U assigned at age 15 or older || 17 || U || 17 years 1 mon 19 days || 17 years 1 mon 19 days || 1 || 38
|-
| 49 || U assigned at age 15 or older || 18 || U || 18 years 1 mon 18 days || 18 years 8 mons 16 days || 6 || 38
|-
| 49 || U assigned at age 15 or older || 32 || U || 32 years 1 mon 21 days || 32 years 5 mons 14 days || 4 || 38
|-
| 49 || U assigned at age 15 or older || 33 || U || 33 years 6 days || 33 years 6 days || 4 || 38
|}

== (#50) There are follow_arrivals where females have a cycle code of <code>n/a</code> ==

=== Problem ===

There are 4,496 follow_arrival rows, for female arriving individuals, that have a sexual cycle code (fa_type_of_cycle) of <code>n/a</code>.

=== Bad data ===

<pre>
select follow_arrival.*
, biography.b_sex
, biography.b_birthdate
from clean.follow_arrival
join clean.biography
on (biography.b_animid = follow_arrival.fa_b_arr_animid)
where biography.b_sex = 'F'
and follow_arrival.fa_type_of_cycle = 'n/a'
order by follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid;
</pre>

=== Solution ===

Change the cycle code to <code>MISS</code>, for these rows. This is done in the <code>clean</code> schema.

This is an indication that future cleanup is required.

== (#51) There are follow_arrivals with invalid fa_data_source values ==

=== Problem ===

There are 546 follow_arrival rows that have fa_data_source values that are not one of:
<code>Tiki</code>
<code>Tiki_Mom</code>
<code>Tiki_ID</code>
<code>Brec</code>.

=== Bad data ===

Query the <code>easy</code> schema, because the data has been cleaned in the <code>clean</code> schema.
<pre>
select follow_arrival.*
from easy.follow_arrival
where follow_arrival.fa_data_source <> 'Tiki'
and follow_arrival.fa_data_source <> 'Tiki_Mom'
and follow_arrival.fa_data_source <> 'Tiki_ID'
and follow_arrival.fa_data_source <> 'Brec'
order by follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid;

-- Summarize with:
select follow_arrival.fa_data_source, count(*)
from easy.follow_arrival
where follow_arrival.fa_data_source <> 'Tiki'
and follow_arrival.fa_data_source <> 'Tiki_Mom'
and follow_arrival.fa_data_source <> 'Tiki_ID'
and follow_arrival.fa_data_source <> 'Brec'
group by follow_arrival.fa_data_source
order by follow_arrival.fa_data_source;
</pre>

=== Solution ===

Change <code>BREC</code> and <code>brec</code> to <code>Brec</code>.

Change <code>TikI</code> to <code>Tiki</code>.

Add the other codes:

fa_data_source count
Tiki_GM 202
Tiki_PM 165
Tiki_SS 22

Also: Tiki_Mom

== (#52) There are follow_arrivals that are almost duplicates ==

=== Problem ===

This entry may end up being multiple problems.

There are follow_arrivals that are near duplicates.
When checking for duplicates on <code>fa_fol_date</code>, <code>fa_fol_b_focal_animid</code>, <code>fa_b_arr_animid</code>
and <code>fa_seq_num</code>, the rows are always unique.
But checking the combination of <code>fa_fol_date</code>, <code>fa_fol_b_focal_animid</code>, <code>fa_b_arr_animid</code>
and <code>fa_time_start</code>,
and <code>fa_time_end</code> yields 184 rows.
Leaving off <code>fa_time_end</code> and just checking the combination of <code>fa_fol_date</code>, <code>fa_fol_b_focal_animid</code>, <code>fa_b_arr_animid</code>
and <code>fa_time_start</code> yields 430 rows.

Why the duplicates?

This entry is a call for a definition of an <code>ARRIVALS</code> row, what does it mean to be a duplicate?

=== Bad data ===

<pre>
-- no duplicates when looking at just sequence number
select fa_fol_date
, fa_fol_b_focal_animid
, fa_b_arr_animid
, fa_seq_num
from clean.follow_arrival
group by fa_fol_date
, fa_fol_b_focal_animid
, fa_b_arr_animid
, fa_seq_num
having count(*) > 1;
</pre>

<pre>
-- Checking against start and end time
with dups as
(select fa_fol_date
, fa_fol_b_focal_animid
, fa_b_arr_animid
, fa_time_start
, fa_time_end
from clean.follow_arrival
group by fa_fol_date
, fa_fol_b_focal_animid
, fa_b_arr_animid
, fa_time_start
, fa_time_end
having count(*) > 1)
select *
from clean.follow_arrival
where exists
(select 1
from dups
where follow_arrival.fa_fol_date = dups.fa_fol_date
and follow_arrival.fa_fol_b_focal_animid
= dups.fa_fol_b_focal_animid
and follow_arrival.fa_b_arr_animid = dups.fa_b_arr_animid
and follow_arrival.fa_time_start = dups.fa_time_start
and follow_arrival.fa_time_end = dups.fa_time_end)
order by follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid
, follow_arrival.fa_time_start
, follow_arrival.fa_time_end
, follow_arrival.fa_seq_num;
</pre>

=== Solution ===

IAN FIXED IN ACCESS 8/24/2016

== (#53) There are follow_arrivals where the arriving chimp does not exist ==

=== Problem ===

There are <s>688</s> <s>84</s> 78 rows, having <s>3</s> <s>6</s> 2 different <code>fa_b_arr_animid</code> values, where the <code>fa_b_arr_animid</code> value is not a <code>biography.b_animid</code> value.

=== Bad Data ===

<pre>
-- The rows
select *
from clean.follow_arrival
where not exists
(select 1
from clean.biography
where biography.b_animid = follow_arrival.fa_b_arr_animid)
order by follow_arrival.fa_b_arr_animid
, follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid;

-- A summary of the bad animal ids
select follow_arrival.fa_b_arr_animid, count(*)
from clean.follow_arrival
where not exists
(select 1
from clean.biography
where biography.b_animid = follow_arrival.fa_b_arr_animid)
group by follow_arrival.fa_b_arr_animid
order by follow_arrival.fa_b_arr_animid;
</pre>

=== Solution ===

ICG fixed in MS Access 10/2025. AMA-->AME, OBE-->POR<br>

SRE: still need to address: UWE, GGl, FAC, FAR, GU, and Sl (as of the April 2026 dump)

FIXED UWE, GGl, FAC, FAR, GU, and Sl IN ACCESS 7/3/2026

UPDATE 2026-07-22: still need to resolve FAR (17 records) and FAC (61 records)

UPDATE 2026-08-18: ICG fixed in Access. FAR-->FAY, FAC-->FIC

== (#54) There are follow_arrivals where females that are too young have a cycle state code that is not <code>0</code>, <code>U</code>, or <code>MISS</code> ==

=== Problem ===

There are <s>26</s> 42 follow_arrival rows, for female arriving individuals, that have a sexual cycle code indicating sexual swelling but are less than 8 years of age.

Note: The test is against the birthdate, not the minimum possible birthdate.

=== Bad data ===

<pre>
select follow_arrival.*, biography.b_sex, biography.b_birthdate
from clean.follow_arrival
join clean.biography
on (biography.b_animid = follow_arrival.fa_b_arr_animid)
where biography.b_sex = 'F'
and follow_arrival.fa_type_of_cycle <> '0'
and follow_arrival.fa_type_of_cycle <> 'U'
and follow_arrival.fa_type_of_cycle <> 'MISS'
and biography.b_birthdate
> (follow_arrival.fa_fol_date
- '8 years'::INTERVAL
)
and follow_arrival.fa_b_arr_animid <> 'MGF'
and follow_arrival.fa_b_arr_animid <> 'MGF2'
and follow_arrival.fa_b_arr_animid <> 'MGF3'
order by follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid;
</pre>

=== Solution ===
IAN TO FIX - CHANGE ALL TO "U"

That works. Would it be better to leave the original MS Access data untouched and have the conversion process make the change in the <code>clean</code> schema?

UPDATE 2026-07-23: The proposed solution has been applied to the clean schema (see commit 06c4178). However, please note that this does not solve Problem #47 and, in fact, forces the fa_type_of_cycle type of records associated with Problem #47 to 'U'.

==(#55) There are community_membership rows that place an individual in a community before birth ==

=== Problem ===

There is one individual who is placed in a community, once, before birth.

Note: The test is against the birthdate, not the minimum possible birthdate.

=== Bad data ===

<pre>
select b.b_animid, b.b_birthdate, cm.cm_start_date
from clean.community_membership as cm
join clean.biography as b
on (b.b_animid = cm.cm_b_animid)
where cm.cm_start_date < b.b_birthdate
order by b.b_animid, cm.cm_start_date;
</pre>

=== Bad solution ===

ICG fixed FN community starte date in MS Access - oct 2025

== (#56) There are follow_arrival focal animids with trailing spaces ==

=== Problem ===

There are 144 follow arrivals where the focal id has trailing spaces.

=== Bad data ===

<pre>
select * from follow_arrival where fa_fol_b_focal_animid <> rtrim(fa_fol_b_focal_animid);
</pre>

=== Solution ===

Remove the trailing spaces in table in the <code>clean</code> schema.

== (#57) GROOM_BOUT duplicate keys ==

=== Problem ===
The data dump says that the <code>GROOM_BOUT</code> table has a primary key consisting of, in order, the columns: <code>GRM_FOL_date</code>, <code>GRM_FOL_B_focal_AnimId</code>, <code>GRM_time_begin</code>, <code>GRM_B_partner_AnimId</code>
But these columns contain duplicate values.

=== Bad Data ===
The duplicate values can be listed (from the <code>raw</code> schema) with:

<pre>SELECT *
FROM raw."GROOM_BOUT"
JOIN (SELECT "GRM_FOL_date" AS the_date
, "GRM_FOL_B_focal_AnimId" AS the_animid
, "GRM_time_begin" AS the_time
, "GRM_B_partner_AnimId" AS the_partner
FROM raw."GROOM_BOUT"
GROUP BY "GRM_FOL_date"
, "GRM_FOL_B_focal_AnimId"
, "GRM_time_begin"
, "GRM_B_partner_AnimId"
HAVING count(*) > 1
) AS gb
ON ("GROOM_BOUT"."GRM_FOL_date" = gb.the_date
AND "GROOM_BOUT"."GRM_FOL_B_focal_AnimId" = gb.the_animid
AND "GROOM_BOUT"."GRM_time_begin" = gb.the_time
AND "GROOM_BOUT"."GRM_B_partner_AnimId" = gb.the_partner);</pre>

=== SOLUTION ===

ICG FIXED IN ACCESS - ALL WERE LEGIT DUPLICATES. REMOVED. 8/18/2026

== (#58) OTHER_SPECIES duplicate keys ==

=== Problem ===
The data dump says that the <code>OTHER_SPECIES</code> table has a primary key consisting of, in order, the columns: <code>OS_FOL_date</code>, <code>OS_FOL_B_focal_AnimID</code>, <code>OS_time_begin</code>
But these columns contain duplicate values.

=== Bad Data ===
The duplicate values can be listed (from the <code>raw</code> schema) with:

<pre>
SELECT *
FROM raw."OTHER_SPECIES"
JOIN (SELECT "OS_FOL_date" AS the_date
, "OS_FOL_B_focal_AnimID" AS the_animid
, "OS_time_begin" AS the_time
FROM raw."OTHER_SPECIES"
GROUP BY "OS_FOL_date"
, "OS_FOL_B_focal_AnimID"
, "OS_time_begin"
HAVING count(*) > 1
) AS os
ON ("OTHER_SPECIES"."OS_FOL_date" = os.the_date
AND "OTHER_SPECIES"."OS_FOL_B_focal_AnimID" = os.the_animid
AND "OTHER_SPECIES"."OS_time_begin" = os.the_time);
</pre>

=== Note ===
ICG fixed in Access 8/18/2026

== (#59) Zero BIOGRAPHY.b_animid_num values ==

=== Problem ===
SokewDB requires that the animal ID number be greater than <code>0</code>, or <code>NULL</code>, but some (what seem to be rows for babys) have a <code>0</code> value.

=== Bad Data ===
The <code>0</code> values can be listed (from the <code>clean</code> schema) with:

<pre>select * from clean.biography where b_animid_num = 0;</pre>

=== Solution ===
Change the <code>0</code> values to <code>NULL</code>.

This is a brute-force, but adequate, solution because it does not validate anything concerning the rows affected.

== (#60) Invalid biography_update_log.made_by values ==

=== Problem ===
There is a <code>biography_update_log.made_by</code> value (<code>SF/EVL</code>) that is not a person. (Not on the <code>PEOPLE</code> table.)

=== Bad Data ===
The invalid values can be listed (from the <code>clean</code> schema) with:

<pre>SELECT *
FROM clean.biography_update_log
WHERE made_by IS NOT NULL
AND NOT EXISTS (SELECT 1
FROM clean.people
WHERE people.person = biography_update_log.made_by);</pre>
=== SOLUTION ===
IAN CHANGED THE SINGLE SF/EVL ENTRY TO EVL IN ACCESS 3/25/2026

''update'': this fix was not reflected in the 2026-07-12 dump.

Codified in clean per commit b793345f4098fa012d559b8f2d01f06d5418126c.
<pre>
-- Problem #60 BIOGRAPHY_UPDATE_LOG.MadeBy contains the invalid combination
-- person code SF/EVL. The source-data resolution is to use EVL.
UPDATE biography_update_log
SET made_by = 'EVL'
WHERE made_by = 'SF/EVL';
</pre>

== * (#61) Invalid follow date/focals pairs in follow_arrival ==

=== Problem ===
There are <code>follow_arrival.fa_fol_b_animid</code> and <code>follow_arrival.fa_fol_date</code> value combinations that do not exist in <code>follow</code>.

=== Bad Data ===
The invalid values can be listed (from the <code>clean</code> schema) with:

<pre>SELECT *
FROM clean.follow_arrival
WHERE NOT EXISTS
(SELECT 1
FROM clean.follow
WHERE follow.fol_date = follow_arrival.fa_fol_date
AND follow.fol_b_animid
= follow_arrival.fa_fol_b_focal_animid)
ORDER BY follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid
, follow_arrival.fa_time_start
, follow_arrival.fa_time_end;</pre>
===SOLUTION===
These are all cases when a juvenile arrival has been extracted from b record notes, but there seems to be no tiki. Do we add WATCHES for these?

== (#62) There are follow_arrival rows with arriving animids with trailing spaces ==

=== Problem ===

There are 39 follow arrivals where the arriving animid has trailing spaces.

=== Bad data ===

<pre>
select * from easy.follow_arrival where fa_b_arr_animid <> rtrim(fa_b_arr_animid);
</pre>

=== Solution ===

Remove the trailing spaces in table in the <code>clean</code> schema. Addressed 9c114ac.

== (#63) There are follow_arrival rows with <code>NULL</code> fa_data_source values ==

=== Problem ===

There are 5,020 follow arrivals where the fa_data_source is <code>NULL</code>.

=== Bad data ===

<pre>SELECT *
FROM easy.follow_arrival
WHERE fa_data_source IS NULL
ORDER BY follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid
, follow_arrival.fa_time_start
, follow_arrival.fa_time_end;</pre>

=== Solution ===
KARL TO FIX
Create a <code>none</code> value in ARRIVAL_SOURCES, and use that value instead of <code>NULL</code> in follow_arrival table in the <code>clean</code> schema. Addressed 8c05232.

== (#64) There are follow_arrival rows with <code>NULL</code> fa_type_of_certainty values ==

=== Problem ===

There is 1 follow arrivals where the fa_type_of_certainty is <code>NULL</code>.

=== Bad data ===

<pre>SELECT *
FROM easy.follow_arrival
WHERE fa_type_of_certainty IS NULL
ORDER BY follow_arrival.fa_fol_date
, follow_arrival.fa_fol_b_focal_animid
, follow_arrival.fa_b_arr_animid
, follow_arrival.fa_time_start
, follow_arrival.fa_time_end;</pre>

=== Solution ===
IAN fixed in Access 3/25/2026

== (#65) There are AGGRESSION_EVENT rows that have "YES" or a space as a ae_bad_observeration_flag value ==

=== Problem ===
There are 3,079 rows where the ae_bad_observation_flag is a space, and 1 row where it is <code>YES</code>.

=== Bad data ===

<pre>SELECT COALESCE(ae.ae_bad_observation_flag, 'NULL'), count(*)
FROM easy.aggression_event AS ae
GROUP BY ae.ae_bad_observation_flag;</pre>

=== Solution ===
KARL: Treat spaces (along with <code>NULL</code>) as <code>FALSE</code>, and <code>YES</code> (along with <code>X</code>() as <code>TRUE</code>.

Stevan: Change the data in the <code>clean</code> schema to contain only boolean values, based on the above.

addressed as part of the solution to problem #74 (commit efa75af037a5e22b2b43675b435f472193a6672f)

== (#66) There are AGGRESSION_EVENT rows that have a space as a ae_decided_flag value ==

=== Problem ===
There are 2,041 rows where the ae_decided_flag is a space.

=== Bad data ===

<pre>SELECT COALESCE(ae.ae_decided_flag, 'NULL'), count(*)
FROM easy.aggression_event AS ae
GROUP BY ae.ae_decided_flag;</pre>

=== Solution ===
KARL: Treat spaces (along with <code>NULL</code>) as <code>FALSE</code>.

Stevan: Fix in the <code>clean</code> schema.

addressed as part of the solution to problem #74 (commit efa75af037a5e22b2b43675b435f472193a6672f)

== (#67) There are AGGRESSION_EVENT rows that have a space or an <code>x</code> as a ae_multiple_aggressor_flag value ==

=== Problem ===
There are 2,397 rows where the ae_multiple_aggressor_flag is a space and 2 rows where the value is an <code>x</code>.

=== Bad data ===

<pre>SELECT COALESCE(ae.ae_multiple_aggressor_flag, 'NULL'), count(*)
FROM easy.aggression_event AS ae
GROUP BY ae.ae_multiple_aggressor_flag;</pre>

=== Solution ===
Treat spaces (along with <code>NULL</code>) as <code>FALSE</code>, and <code>x</code> (along with <code>X</code>) as <code>TRUE</code>.

addressed as part of the solution to problem #74 (commit efa75af037a5e22b2b43675b435f472193a6672f)

== (#68) There are AGGRESSION_EVENT rows that have a space as a ae_multiple_recipient_flag value ==

=== Problem ===
There are 2,528 rows where the ae_multiple_recipient_flag is a space.

=== Bad data ===

<pre>SELECT COALESCE(ae.ae_multiple_recipient_flag, 'NULL'), count(*)
FROM easy.aggression_event AS ae
GROUP BY ae.ae_multiple_recipient_flag;</pre>

=== Solution ===
Treat spaces (along with <code>NULL</code>) as <code>FALSE</code>.

addressed as part of the solution to problem #74 (commit efa75af037a5e22b2b43675b435f472193a6672f)

== (#69) There are duplicate year*community records in AGGRESSION_EVENT_LOG ==

=== Problem ===
There are 11 duplicate pairs of year*community records in AGGRESSION_EVENT_LOG. There can be, at most, one row per-community, per-year. The duplicates each have a `b_rec_english` value of `ALL` or `F-F (ALL); F-M (ALL); M-M (ALL)`.

=== Bad data ===

<pre>
SELECT *
FROM clean.aggression_event_log
JOIN (
SELECT
year,
community
FROM
clean.aggression_event_log
GROUP BY
year,
community
HAVING count(*) > 1
) AS dups ON (
dups.year = aggression_event_log.year
AND dups.community = aggression_event_log.community
)
order by
aggression_event_log.year,
aggression_event_log.community ;
</pre>

=== Solution ===

IAN fixed 8/18/2026

== (#70) There are AGGRESSION_EVENT rows that do not have a matching FOLLOW for the same focal ID and date. ==

=== Problem ===
There are 1494 records where clean.aggression_event does not have a matching row in clean.follow for the same focal ID and date.

=== Bad data ===

<pre>
WITH problem_rows AS (
SELECT
ae.*,
EXISTS (
SELECT 1
FROM clean.follow f_id
WHERE f_id.fol_b_animid = ae.ae_fol_b_focal_id
) AS focal_id_exists_in_follow
FROM clean.aggression_event ae
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
)
SELECT
pr.ae_date,
pr.ae_time,
pr.ae_fol_b_focal_id,
pr.ae_b_aggressor_id,
pr.ae_b_recipient_id,
pr.ae_source,
pr.ae_full_description,
CASE
WHEN pr.focal_id_exists_in_follow THEN 'missing_follow_on_same_date'
ELSE 'focal_id_not_found_in_follow'
END AS issue_type
FROM problem_rows pr
ORDER BY
pr.ae_date,
pr.ae_fol_b_focal_id,
pr.ae_time;
</pre>

=== Resolution ===

Resolved by commit `1cc9e694a1f6a53c396d9074f77d5d279018721b`.

A matching FOLLOW row is not required for an aggression event. WATCHES
represents the observation context independently of FOLLOW. A B-record
WATCHES row may represent either a follow or an ad-hoc observation such as
an aggression.

During conversion, `load_aggressions.m4` looks for an existing B-record
WATCHES row having the aggression event's focal individual and date. If no
such row exists, the loader creates one using the aggression event's focal,
community, and date. The aggression's EVENTS row is then related to that
WATCHES row.

The original diagnostic query will continue to identify aggression records
without matching FOLLOW rows. These rows are not necessarily bad data and
are not expected to disappear.

== * (#71) There are AGGRESSION_EVENT animal ids that are not reflected in among FOLLOW animal ids. ==

=== Problem ===
There are 16 unique(!) aggression_event.ae_fol_b_focal_id records that are not reflected among follow.fol_b_animid values.

=== Processing note ===

All of the problematic records are reflected in the rows excluded as part of
Problem #70. As such, there is not a separate exclusion clause in the conversion
process for these data. As a corollary, addressing all issues in Problem #70
would address all Problem #71 infractions well.

Problem #70 concerns aggression rows without a matching follow for the same
focal ID and date. Canonical master solves that relationship problem by creating
a B-record WATCHES row when no suitable watch exists. An aggression therefore
does not inherently require a FOLLOW.

Problem #71 is a stricter subset: its focal IDs never occur among follow focal
IDs. More importantly, a read-only check of the local data found that the
documented 16 IDs are also absent from clean.biography, including values such as
DL DUK, GA GGL, GROUP, Males, and stranger.

<pre>WATCHES.AnimID → BIOGRAPHY_DATA.AnimID</pre>

Consequently:

- If a #71 focal ID is a valid biography ID, the #70 WATCHES solution handles it.
- If it is not a valid biography ID, creating the WATCHES row fails its foreign key.
- The documented 16 IDs appear to be in the latter category and therefore require a separate source-data decision or clean-stage mapping.

There is also a branch-specific complication: the temporary branch loader still
has the old #70 WHERE EXISTS filter. It excludes all rows lacking a matching
follow, including every #71 row. Thus the successful mung conversion does not
demonstrate that these rows were converted. Canonical master does not have that
filter and would expose invalid #71 focal IDs when attempting to create their
WATCHES rows.

Therefore, the Problem #71 processing note is too broad. The #70 solution
removes the requirement for a corresponding follow, but it does not resolve
invalid or composite focal animal IDs. Problem #71 should remain a separate
unresolved data-quality problem, and mung’s stale #70 exclusion should
eventually be removed when testing the full solution.

=== Bad data ===

<pre>
SELECT
ae_fol_b_focal_id,
COUNT(*) AS row_count
FROM clean.aggression_event
WHERE ae_fol_b_focal_id NOT IN (
SELECT DISTINCT fol_b_animid
FROM clean.follow
WHERE fol_b_animid IS NOT NULL
)
GROUP BY ae_fol_b_focal_id
ORDER BY
row_count DESC,
ae_fol_b_focal_id
;
</pre>

=== SOLUTION ===

IAN FIXED IN ACCESS 8/25/2026

== (#72) There are AGGRESSION_EVENT recipient certainty flags other than `Y` or `N` (required). ==

=== Problem ===
There are 28,322 AGGRESSION_EVENT rows that have a ae_recipient_certainty_flag other than `Y` or `N` as required.

=== Bad data ===

==== Full record ====

<pre>
SELECT
ae.*,
COALESCE(ae_recipient_certainty_flag, '<NULL>') AS raw_flag
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND UPPER(BTRIM(COALESCE(ae.ae_recipient_certainty_flag, ''))) NOT IN ('N', 'Y') ;
</pre>

==== Summary of non-compliant values ====

<pre>
SELECT
COALESCE(ae_recipient_certainty_flag, '<NULL>') AS raw_flag,
COUNT(*) AS row_count
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND UPPER(BTRIM(COALESCE(ae.ae_recipient_certainty_flag, ''))) NOT IN ('N', 'Y')
GROUP BY COALESCE(ae_recipient_certainty_flag, '<NULL>')
ORDER BY row_count DESC, raw_flag;
</pre>

=== Solution ===

Ian or Elizabeth to please confirm.

* sokwe conversions:
** ' ' to '0'
** 'N' to '0'
** 'NO' to '0'
** 'X' to '1'
** 'Y%' to '1'

ICG AND ELO CONFIRM 7/3/2026

KARL

Stevan: The usual. Fix in the <code>clean</code> schema.

addressed as part of the solution to problem #74 (commit efa75af037a5e22b2b43675b435f472193a6672f)

== (#73) There are AGGRESSION_EVENT aggression_event.ae_b_recipient_id values that are NULL. ==

=== Problem ===
There are 5,820 AGGRESSION_EVENT rows for which the aggression_event.ae_b_recipient_id is NULL.

=== Bad data ===

<pre>
SELECT
ae.ae_date,
ae.ae_time,
ae.ae_fol_b_focal_id,
ae.ae_b_aggressor_id,
ae.ae_b_recipient_id,
ae.ae_recipient_certainty_flag,
ae.ae_full_description,
ae.ae_source,
ae.ae_comments
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND ae.ae_b_recipient_id IS NULL
ORDER BY ae.ae_date, ae.ae_fol_b_focal_id, ae.ae_time;
</pre>

=== SOLUTION ===

CAN THESE ALL BE NULL? IF NOT, THEN WE NEED TO DO SOMETHING ABOUT INSTANCES WHEN 'GROUP' IS THE TARGET.

==== Karl's comment ====

The usual solution is to have the unknown individual, <code>UNK</code>, be the target. If NULL has some other special meaning, other than we don't know at whom the aggression was directed, we could make up another "special" <code>BIOGRAPHY_DATA</code> row, similar to the <code>UNK</code>, <code>MGF</code>, <code>MGM</code>, etc. rows.

Yes, use UNK.

== (#74) There are AGGRESSION_EVENT event flags (multiple columns) that have values other than X or NULL. ==

=== Problem ===

Among AGGRESSION_EVENT flags (ae_bad_observation_flag, ae_bristle_flag, ae_chase_flag, ae_contact_flag, ae_contact_flag, ae_decided_flag, ae_display_flag, ae_multiple_aggressor_flag, ae_multiple_recipient_flag, ae_vocal_flag, ae_vocal_flag) there are 2,864 rows that have a value other than `X` or NULL (required) for one or more of the flags.

=== Bad data ===

==== Full record ====

<pre>
SELECT
ae.ae_date,
ae.ae_time,
ae.ae_fol_b_focal_id,
ae.ae_b_aggressor_id,
ae.ae_b_recipient_id,
ae.ae_decided_flag,
ae.ae_multiple_aggressor_flag,
ae.ae_multiple_recipient_flag,
ae.ae_bad_observation_flag,
ae.ae_bristle_flag,
ae.ae_display_flag,
ae.ae_chase_flag,
ae.ae_contact_flag,
ae.ae_vocal_flag,
ae.ae_source,
ae.ae_comments
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND ae.ae_b_recipient_id IS NOT NULL
AND (
(ae.ae_decided_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_decided_flag)) <> 'X')
OR (ae.ae_multiple_aggressor_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_multiple_aggressor_flag)) <> 'X')
OR (ae.ae_multiple_recipient_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_multiple_recipient_flag)) <> 'X')
OR (ae.ae_bad_observation_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_bad_observation_flag)) <> 'X')
OR (ae.ae_bristle_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_bristle_flag)) <> 'X')
OR (ae.ae_display_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_display_flag)) <> 'X')
OR (ae.ae_chase_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_chase_flag)) <> 'X')
OR (ae.ae_contact_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_contact_flag)) <> 'X')
OR (ae.ae_vocal_flag IS NOT NULL AND UPPER(BTRIM(ae.ae_vocal_flag)) <> 'X')
)
ORDER BY ae.ae_date, ae.ae_fol_b_focal_id, ae.ae_time;
</pre>

==== Summary of non-compliant values ====

<pre>
WITH base AS (
SELECT ae.*
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND ae.ae_b_recipient_id IS NOT NULL
),
flag_values AS (
SELECT 'ae_decided_flag' AS flag_name, COALESCE(ae_decided_flag, '<NULL>') AS raw_value FROM base
UNION ALL
SELECT 'ae_multiple_aggressor_flag', COALESCE(ae_multiple_aggressor_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_multiple_recipient_flag', COALESCE(ae_multiple_recipient_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_bad_observation_flag', COALESCE(ae_bad_observation_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_bristle_flag', COALESCE(ae_bristle_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_display_flag', COALESCE(ae_display_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_chase_flag', COALESCE(ae_chase_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_contact_flag', COALESCE(ae_contact_flag, '<NULL>') FROM base
UNION ALL
SELECT 'ae_vocal_flag', COALESCE(ae_vocal_flag, '<NULL>') FROM base
)
SELECT
flag_name,
raw_value,
COUNT(*) AS row_count
FROM flag_values
WHERE raw_value <> '<NULL>'
AND UPPER(BTRIM(raw_value)) <> 'X'
GROUP BY flag_name, raw_value
ORDER BY flag_name, row_count DESC, raw_value;
</pre>

=== Solution ===

Ian or Elizabeth to please confirm.

* clean schema conversions (mimic patterns in Access):
** `?` to ' '
** `Y%` to `X`
** `X%` to `X`

* sokwe conversions:
** ' ' to '0'
** 'X' to '1'

ICG AND ELO CONFIRM 7/3/26

addressed with commits efa75af037a5e22b2b43675b435f472193a6672f and 921b5ca3e9ca3d1e06fe189b4775aae7dcc484bf
note that the solution to problem #74 was an umbrella fix that also addressed #65-68, and #72

== * (#75) There are AGGRESSION_EVENT records where ae_extracted_by values are not in the people table. ==

=== Problem ===

There are 28,513 AGGRESSION_EVENT records where ae_extracted_by does not match a person in the PEOPLE table.

=== Bad data ===

==== Full record ====

<pre>
WITH scoped AS (
SELECT ae.*
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND ae.ae_b_recipient_id IS NOT NULL
)
SELECT
s.ae_date,
s.ae_time,
s.ae_fol_b_focal_id,
s.ae_b_aggressor_id,
s.ae_b_recipient_id,
s.ae_extracted_by,
s.ae_source,
s.ae_full_description,
s.ae_comments
FROM scoped s
WHERE NOT EXISTS (
SELECT 1
FROM people p
WHERE p.person = BTRIM(COALESCE(s.ae_extracted_by, ''))
OR LOWER(p.name) = LOWER(BTRIM(COALESCE(s.ae_extracted_by, '')))
)
ORDER BY s.ae_date, s.ae_fol_b_focal_id, s.ae_time;
</pre>

==== Summary of non-compliant values ====

<pre>
WITH scoped AS (
SELECT ae.*
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND ae.ae_b_recipient_id IS NOT NULL
)
SELECT
BTRIM(COALESCE(ae_extracted_by, '')) AS raw_extractedby,
COUNT(*) AS row_count
FROM scoped s
WHERE NOT EXISTS (
SELECT 1
FROM people p
WHERE p.person = BTRIM(COALESCE(s.ae_extracted_by, ''))
OR LOWER(p.name) = LOWER(BTRIM(COALESCE(s.ae_extracted_by, '')))
)
GROUP BY BTRIM(COALESCE(ae_extracted_by, ''))
ORDER BY row_count DESC, raw_extractedby;
</pre>

=== SOLUTION ===

ALLOW FOR NOW. NOT SURE IF/HOW THE PEOPLE TABLE IS POPULATED.

==== Karl's comment ====

We populate the <code>PEOPLE</code> however we like. We just make things up.
We can add all these values to the table as different people.
However, unless you set the <code>PEOPLE.Active</code> to <code>FALSE</code>
you run the risk of future data entry using a bunch of odd or outdated people.

== (#76) There are AGGRESSION_EVENT rows where the Actor/Actee are not in the biography table. ==

=== Problem ===

There are 10,866 combined (but see following!) rows where the ae_b_aggressor_id and/or ae_b_recipient_id is not in the BIOGRAPHY table.

Notes:
* The row count is inflated by a CROSS LATERAL JOIN, which yields the number of combined, pivoted records for both ae_b_aggressor_id and ae_b_recipient, not the actual number of confounding AGGRESSION_EVENT rows.
* The queries currently exclude ae_b_recipient_id values that are NULL, which seems a related but separte problem; the number of rows jumps to 10,357 if ae_b_recipient_id = NULL are included.
* How are we treating ae_b_recipient_id values such as `group`, `males`, `females`, etc.?
* Whitespace is considered in the incongruence such that, for example, `VIN ` (with a space) does not match `VIN` in biography.

=== Bad data ===

==== Full record ====

<pre>
SELECT
ae.ae_date,
ae.ae_time,
ae.ae_fol_b_focal_id,
v.role_name,
v.participant,
ae.ae_b_aggressor_id,
ae.ae_b_recipient_id,

ae.ae_source,
ae.ae_full_description,
ae.ae_comments
FROM clean.aggression_event ae
CROSS JOIN LATERAL (
VALUES
('Actor'::text, ae.ae_b_aggressor_id),
('Actee'::text, ae.ae_b_recipient_id)
) AS v(role_name, participant)
WHERE v.participant IS NOT NULL
AND v.participant <> ''
AND NOT EXISTS (
SELECT 1
FROM clean.biography b
WHERE b.b_animid = v.participant
)
ORDER BY ae.ae_date, ae.ae_fol_b_focal_id, ae.ae_time, v.role_name;
</pre>

==== Summary of non-compliant values ====

''trimmed_match'' denotes a match between Actor/Actee and biography if whitespace is trimmed.

<pre>
SELECT
v.role_name,
v.participant AS raw_participant,
b_trimmed.b_animid AS trimmed_match,
COUNT(*) AS row_count
FROM clean.aggression_event ae
CROSS JOIN LATERAL (
VALUES
('Actor'::text, ae.ae_b_aggressor_id),
('Actee'::text, ae.ae_b_recipient_id)
) AS v(role_name, participant)
LEFT JOIN clean.biography b_trimmed
ON BTRIM(v.participant) = BTRIM(b_trimmed.b_animid)
WHERE v.participant IS NOT NULL
AND v.participant <> ''
AND NOT EXISTS (
SELECT 1
FROM clean.biography b
WHERE b.b_animid = v.participant
)
GROUP BY v.role_name, v.participant, b_trimmed.b_animid
ORDER BY row_count DESC, v.role_name, v.participant;
</pre>

=== SOLUTION ===

ICG fixed in Access 7/7/2026. replaced "group", "males", etc with UNK. Fixed bad IDs. Updated comments field to reflect original entry terms.

==== Karl comment ====

You can't just "allow them". You can make rows in BIOGRAPHY_DATA that have AnimID values that correspond to the values that appear. It's hard to say whether or not these rows would have to be "special". Probably not, because anybody can aggress against anybody. But if you're going to make a whole bunch of "dummy" individuals (138 of them, to be exact), it might be prudent to add a flag to BIOGRAPHY_DATA that says whether or not the row corresponds to a specific chimpanzee.
(That flag could even be used to filter the dummy individuals out of the BIOGRAPHY view.)

== (#77) There are AGGRESSION_EVENT rows where ae_recipient_behavior is NULL or an empty string. ==

=== Problem ===

There are 9,102 records where ae_recipient_behavior is NULL or an empty string.

=== Bad data ===

<pre>
SELECT *
FROM clean.aggression_event
WHERE
ae_recipient_behavior IS NULL
OR ae_recipient_behavior = '' ;
</pre>

=== SOLUTION ===

ALLOW FOR NOW

==== Karl's comments ====

Change the NULL values to the empty string in the <code>clean</code> schema.
Addressed with commit a84462b47c8b2100caa02958623817981179d068.

== (#78) There are AGGRESSION_EVENT rows where ae_full_description is NULL or an empty string. ==

=== Problem ===

There are 4,718 records where ae_full_description is NULL or an empty string.

=== Bad data ===

<pre>
SELECT *
FROM clean.aggression_event
WHERE
ae_full_description IS NULL
OR ae_full_description = '' ;
</pre>

=== SOLUTION ===

ALLOW FOR NOW.

==== Karl's comments ====

Change the NULL values to the empty string in the clean schema.
Addressed with commit a84462b47c8b2100caa02958623817981179d068.

== (#79) There are AGGRESSION_EVENT participants outside their valid study participation window in the biography data. ==

=== Problem ===

There are <s>46</s> 6 records where an Actor and/or Actee are in the AGGRESSION_EVENT table but outside their valid study participation window in the biography data. Note that the query strips white space when comparing the ids of aggression participant ids and biography animid ids given that the focus here is on checking for events outside of defined date ranges rather than matching ids and assuming that issues concerning white space will be resolved (also in the migration work-around). The query further winnows aggression events for which there is a valid follow record (not in the migration work-around).

=== Bad data ===

<pre>
WITH candidate_agg AS (
SELECT
ae.ae_date,
ae.ae_time,
ae.ae_fol_b_focal_id,
BTRIM(COALESCE(ae.ae_b_aggressor_id, '')) AS actor_id,
BTRIM(COALESCE(ae.ae_b_recipient_id, '')) AS actee_id,
ae.ae_source,
ae.ae_full_description
FROM clean.aggression_event ae
WHERE EXISTS (
SELECT 1
FROM clean.follow f
WHERE f.fol_b_animid = ae.ae_fol_b_focal_id
AND f.fol_date = ae.ae_date
)
AND ae.ae_b_recipient_id IS NOT NULL
AND EXISTS (
SELECT 1
FROM clean.biography b
WHERE b.b_animid = BTRIM(COALESCE(ae.ae_b_aggressor_id, ''))
)
AND EXISTS (
SELECT 1
FROM clean.biography b
WHERE b.b_animid = BTRIM(COALESCE(ae.ae_b_recipient_id, ''))
)
),
participants AS (
SELECT
c.ae_date,
c.ae_time,
c.ae_fol_b_focal_id,
'Actor' AS role,
c.actor_id AS participant,
c.ae_source,
c.ae_full_description
FROM candidate_agg c
UNION ALL
SELECT
c.ae_date,
c.ae_time,
c.ae_fol_b_focal_id,
'Actee' AS role,
c.actee_id AS participant,
c.ae_source,
c.ae_full_description
FROM candidate_agg c
)
SELECT
p.ae_date,
p.ae_time,
p.ae_fol_b_focal_id,
p.role,
p.participant,
b.b_entrydate,
b.b_departdate,
CASE
WHEN p.ae_date < b.b_entrydate THEN 'before_entry'
WHEN p.ae_date > b.b_departdate THEN 'after_departure'
ELSE 'ok'
END AS violation_type,
p.ae_source,
p.ae_full_description
FROM participants p
JOIN clean.biography b
ON b.b_animid = p.participant
WHERE p.ae_date < b.b_entrydate
OR p.ae_date > b.b_departdate
ORDER BY p.ae_date, p.ae_fol_b_focal_id, p.ae_time, p.role, p.participant;
</pre>

=== SOLUTION ===
ICG FIXED IN ACCESS 7/6/2026

==== Karl comment ====
There is no good way to "allow for now".

We could change the under study dates (EntryDate, DepartDate) of the affected individuals "for now". That would "solve" the problem.

Or just don't convert the data and let the users get the info from the <code>clean</code> schema and integrate it into their processing.

Update 2026-07-26: row count is much improved but there are six records that violate this constraint.

== (#80) There are AGGRESSION_EVENT rows where ae_aggressor_behavior is NULL or an empty string. ==

=== Problem ===

There are 143 records where ae_aggressor_behavior is NULL or an empty string.

=== Bad data ===

<pre>
SELECT *
FROM clean.aggression_event
WHERE ae_aggressor_behavior IS NULL
OR BTRIM(ae_aggressor_behavior) = ''
ORDER BY ae_date, ae_fol_b_focal_id, ae_time;
</pre>

=== SOLUTION ===

ALLOW FOR NOW. EVENTUALLY A VOLUNTEER CAN POPULATE USING THE FULL DESCRIPTION FIELD.

Change the <code>NULL</code> values to the empty string in the <code>clean</code> schema.
Addressed with commit a84462b47c8b2100caa02958623817981179d068.

== (#81) There are AGGRESSION_EVENT rows where ae_time is NULL or outside the allowable window ('04:00:00', '20:00:00'). ==

=== Problem ===

There are 112 AGGRESSION_EVENT rows where ae_time is NULL or outside the allowable window ('04:00:00', '20:00:00').

=== Bad data ===

<pre>
SELECT
ae_date,
ae_time,
ae_fol_b_focal_id,
ae_b_aggressor_id,
ae_b_recipient_id,
ae_source,
ae_full_description,
CASE
WHEN ae_time IS NULL THEN 'null_time'
WHEN ae_time < '04:00:00'::time THEN 'below_min'
WHEN ae_time > '20:00:00'::time THEN 'above_max'
END AS time_issue
FROM clean.aggression_event
WHERE ae_time IS NULL
OR ae_time < '04:00:00'::time
OR ae_time > '20:00:00'::time
ORDER BY ae_date, ae_fol_b_focal_id, ae_time;
</pre>

=== SOLUTION ===
MOST ARE NULL VALUES BECAUSE THEY WERE 'BOOK EXTRACTS'. ASSIGN A 'DUMMY' TIME?

ICG FIXED THE FEW THAT WERE NOT BETWEEN 4:00 AND 20:00 IN ACCESS 7/6/2026

==== Karl's comment ====

Allowing means changing the "too early" and "too late" limits for all times, appearing anywhere in the db. This is easy, but opens the door for other data errors.

The change to the time limits cannnot (easily) be restricted to just the table that is converted. Well.... It can. But the change will not be reflected in the documentation.
(And it'll be completely annoying because a change will have to be re-made each time the conversion program is re-run, which is a lot because we have a lot of errors to resolve.)

Changing the limits back is less than easy. A separate SQL statement needs to be run for each time column that appears in the database. Probably better is to dump the database content, rebuild the database tables, etc., and then re-load the database content.
If any time values are out of bounds, the re-load of the database content will fail.

Executive Summary: Anything's possible but you're coloring outside the lines here.

=== resolution ===

The schema already has a deliberate non-NULL representation for an unknown
aggression time: 00:00, defined as sdb_no_time.

The design is explicit:

- EVENTS.Start and EVENTS.Stop remain NOT NULL.
- Normal event times must be between 04:00 and 20:00.
- Aggressions alone may use 00:00 to mean “no time was recorded.”
- Both start and stop must be 00:00 together.

Existing implementation Commit 37bb5493b7571084e6d57aa45c5440dcb0db357e Karl O.
Pinc 2026-07-15

== (#82) There are AGGRESSION_EVENT rows where severity (ae_fight_category) is not an allowable value. ==

=== Problem ===

There are 2,041 AGGRESSION_EVENT rows where severity (ae_fight_category) is not an allowable value [('unrated', '0', '1', '2', '?')]. Note that all non-compliant values are NULL or some variation of white space.

=== Bad data ===

==== Full record ====

<pre>
SELECT
ae_date,
ae_time,
ae_fol_b_focal_id,
ae_b_aggressor_id,
ae_b_recipient_id,
ae_fight_category,
BTRIM(COALESCE(ae_fight_category, '')) AS normalized_fight_category,
ae_source,
ae_full_description,
ae_comments
FROM clean.aggression_event
WHERE ae_fight_category IS NOT NULL
AND (
BTRIM(ae_fight_category) = ''
OR BTRIM(ae_fight_category) NOT IN ('unrated', '0', '1', '2', '?')
)
ORDER BY ae_date, ae_fol_b_focal_id, ae_time;
</pre>

==== Summary of non-compliant values ====

<pre>
SELECT
BTRIM(COALESCE(ae_fight_category, '')) AS offending_value,
COUNT(*) AS row_count
FROM clean.aggression_event
WHERE ae_fight_category IS NOT NULL
AND (
BTRIM(ae_fight_category) = ''
OR BTRIM(ae_fight_category) NOT IN ('unrated', '0', '1', '2', '?')
)
GROUP BY BTRIM(COALESCE(ae_fight_category, ''))
ORDER BY row_count DESC, offending_value;
</pre>

==== Fine-grain assessment of non-compliant values ====

Note the ''white space'' problem.

<pre>
select distinct '"' || ae_fight_category || '"' from clean.aggression_event ;
</pre>

=== SOLUTION ===
ALLOW FOR NOW - MOST DATA ENTERERS DIDN'T FILL THIS COLUMN IN

==== Karl's comment ====

There seems to be only a problem because NULL values exist. The other non-conformant data seems to be gone from MS Access. So make a fight category code that is "no data" and clean up the data in the <code>clean</code> schema.

NULL means unrated

=== resolution ===

Added `NODATA` to `AGG_SEVERITIES` for aggression records where no
fight-category data were recorded. This is distinct from `unrated`, which means
the aggression was not evaluated for severity.

During construction of the `clean` schema, `ae_fight_category` values are
trimmed. `NULL`, empty-string, and whitespace-only values are converted to
`NODATA`. The aggression loader then copies the cleaned value directly into
`AGGRESSIONS.Severity`, allowing unexpected nonblank codes to remain visible as
conversion errors.

Implemented in commit `d911ef2` (`Resolve Problem #82 with NODATA severity`). A
complete conversion using the updated code succeeded.

== * (#83) There are AGGRESSION_EVENT rows where the Actor and Actee have the same animal ID. ==

=== Problem ===

There are 1,832 <s>363</s> AGGRESSION_EVENT rows where `ae_b_aggressor_id` and `ae_b_recipient_id` share the same animal id.

=== Bad data ===

<pre>
WITH potential_role_dupes AS (
SELECT
ae.ae_date,
ae.ae_time,
ae.ae_fol_b_focal_id,
ae.ae_b_aggressor_id,
ae.ae_b_recipient_id,
ae.ae_source,
ae.ae_full_description,
ae.ae_comments,
ae.dup
FROM clean.aggression_event ae
WHERE ae.ae_b_recipient_id IS NOT NULL
AND ae.ae_b_aggressor_id IS NOT NULL
)
SELECT
ae_date,
ae_time,
ae_fol_b_focal_id,
ae_b_aggressor_id,
ae_b_recipient_id,
ae_source,
ae_full_description,
ae_comments,
dup
FROM potential_role_dupes
WHERE ae_b_aggressor_id = ae_b_recipient_id
ORDER BY ae_date, ae_fol_b_focal_id, ae_time;
</pre>

=== Processing note ===

This data problem generates the following error but note that, in practice, this error could be generated for other reasons as well.

<pre>
./load_chunks.sh load_aggressions.m4 clean.aggression_event
psql:<stdin>:394: ERROR: duplicate key value violates unique constraint "On ROLES, Participant + EID must be unique"
DETAIL: Key (participant, eid)=(FD, 759513) already exists.
CONTEXT: SQL statement "INSERT INTO roles (
eid
, role
, participant)
VALUES (
CURRVAL('events_eid_seq')
, 'Actee'
, this_ae.ae_b_recipient_id)"
PL/pgSQL function inline_code_block line 186 at SQL statement
make: *** [Makefile:373: load_aggressions] Error 3
</pre>

=== SOLUTION ===
ALLOW FOR NOW. EVENTUALLY A VOLUNTEER CAN CHECK AND FIX

==== Karl's comment ====

If we allow, we allow for all event types, not just aggressions. Groomings, matings, etc would all allow "self-dealing". I'd rather not.

How about not converting this data and letting people who want it get it from the <code>clean</code> schema?

do not convert these rows, fix them later

== (#84) Some follows have a community with trailing spaces ==

=== Problem ===

There are 3 follows where the fol_cl_community_id has trailing spaces.

=== Bad Data ===

Cleaned in <code>clean</code> schema, so query <code>easy</code> schema.

<pre>
select '"' || fol_cl_community_id || '"' AS fol_cl_community_id_untrimmed,
*
FROM easy.follow
WHERE RTRIM(fol_cl_community_id) <> fol_cl_community_id;
</pre>

=== SOLUTION ===

KARL TO FIX

Fix by trimming spaces in the <code>clean</code> schema.

== (#85) There are numerous instances of local food names that translate to multiple scientific food names. ==

=== Problem ===

There are 47 records documenting instances where a local food name (fl_local_food_name) is associated with more than one food scientific food name (fl_sci_food_name).

=== Bad Data ===

<pre>
WITH duplicates AS (
SELECT
fl_sci_food_name,
COUNT(*) AS count
FROM clean.food_lookup
GROUP BY fl_sci_food_name
HAVING COUNT(*) > 1
)
SELECT
fl_local_food_name,
duplicates.fl_sci_food_name
FROM clean.food_lookup
JOIN duplicates ON clean.food_lookup.fl_sci_food_name = duplicates.fl_sci_food_name
ORDER BY fl_sci_food_name;
</pre>

=== Notes ===

But see problem #87. This problem is only relevant if codes.food_names.description is derived from fl_sci_food_name (i.e., not fl_sci_food_name_gen); else, refer to #87.

=== SOLUTION ===
ALLOW FOR NOW, BUT NEEDS TO BE DISCUSSED AS A PI GROUP

9/3/2026 - ICG: Actually this returns cases where a scientific food name is associated with more than one local food name. That is, There can be multiple words for the same latin name.

==== Karl's comment ====

To allow, do some hackery when constructing the support tables. Make non-unique values unique by adding extra "duplicate #1 --" sort of text into the value of the column. Either "by hand" in the conversion code or by way of some algorythm.

==== 2026-09-13 conversion decision ====

<code>FOOD_NAMES.Description</code> will be derived from
<code>fl_sci_food_name_gen</code>, not <code>fl_sci_food_name</code>. Problem
#87 therefore controls description collisions for this conversion.

The historical query above does not test the relationship stated in the
heading: it finds a scientific name shared by multiple local names. Do not use
the historical count of 47 as a conversion assertion. If this issue is
revisited, first replace the query with one that tests the intended
relationship against the refreshed source.

== (#86) There are numerous instances in the food_part_lookup that that conflate name, initial, and/or the english translation. ==

=== Problem ===

There are numerous instances in the food_part_lookup that that conflate name, initial, and/or the english translation. These issues are detailed below. Please note that this problem concerns the food_part_lookup table specifically.

Related, we need to clarify how we are treating the initials of food parts. Currently, initials are added to the codes.food_parts table with the corresponding food part name but with a trailing `-initial` so as not to violate the
codes.food_parts.description unique constraint. So, both the food part name and food part initials are included in food events. This is probably not a good approach. Better would be to map the initial to the corresponding food part name and use only the full name (but then you do lose some reference to the original data). Ian needs to please consider.

=== Bad Data ===

==== query ====

<pre>
WITH source_rows AS (
SELECT
ROW_NUMBER() OVER () AS src_ord,

COALESCE((
SELECT ARRAY_AGG(tok)
FROM (
SELECT BTRIM(x) AS tok
FROM REGEXP_SPLIT_TO_TABLE(COALESCE(fpl_local_food_part, ''), E'[;:,]') AS x
WHERE BTRIM(x) <> ''
) s
), ARRAY[]::text[]) AS local_arr,

COALESCE((
SELECT ARRAY_AGG(tok)
FROM (
SELECT BTRIM(x) AS tok
FROM REGEXP_SPLIT_TO_TABLE(COALESCE(fpl_food_part_initials, ''), E'[;:,]') AS x
WHERE BTRIM(x) <> ''
) s
), ARRAY[]::text[]) AS initials_arr,

COALESCE((
SELECT ARRAY_AGG(tok)
FROM (
SELECT BTRIM(x) AS tok
FROM REGEXP_SPLIT_TO_TABLE(COALESCE(fpl_english_food_part, ''), E'[;:,]') AS x
WHERE BTRIM(x) <> ''
) s
), ARRAY[]::text[]) AS english_arr

FROM clean.food_part_lookup
),
expanded AS (
SELECT
s.src_ord,
gs.idx,
COALESCE(s.local_arr[gs.idx], '') AS fpl_local_food_part,
COALESCE(s.initials_arr[gs.idx], '') AS fpl_food_part_initials,
COALESCE(s.english_arr[gs.idx], '') AS fpl_english_food_part
FROM source_rows s
CROSS JOIN LATERAL GENERATE_SERIES(
1,
GREATEST(
CARDINALITY(s.local_arr),
CARDINALITY(s.initials_arr),
CARDINALITY(s.english_arr)
)
) AS gs(idx)
),
deduped AS (
SELECT
e.*,
ROW_NUMBER() OVER (
PARTITION BY
e.fpl_local_food_part,
e.fpl_food_part_initials,
e.fpl_english_food_part
ORDER BY e.src_ord, e.idx
) AS rn
FROM expanded e
)
SELECT
fpl_local_food_part,
fpl_food_part_initials,
fpl_english_food_part
FROM deduped
WHERE rn = 1
ORDER BY src_ord, idx
fpl_local_food_part,
fpl_food_part_initials,
fpl_english_food_part;
</pre>

==== summary ====

{| class="wikitable"
! fpl_local_food_part !! fpl_food_part_initials !! fpl_english_food_part
|-
| CHIPUKIZA || C || SHOOTS
|-
| MAJANI || J || LEAVES
|-
| MBEGU || MB || SEEDS
|-
| WADUDU || W || INSECTS
|-
| MABUA || B || PITH
|-
| MAGOMA || G || BARK
|-
| MATUNDA || T || FRUIT
|-
| MAUA || M || FLOWERS
|-
| UTOMVI || U || SAP
|-
| WADUDU WENGINE || D || INSECTS
|-
| UTOMVU || U || SAP
|-
| MCHWA || W || TERMITES
|-
| MIFUPA || NA || BONES
|-
| MITI || NA || TREE
|-
| MIZIZI || NA || ROOTS
|-
| NA || NA || NOT APPLICABLE
|-
| NONE || NA || None
|-
| NYAMA || N || MEAT
|-
| SIAFU || S || INSECTS
|-
| UNRECORDED || NA || UNRECORDED
|-
| WADUDU || D || INSECTS
|}

==== notes ====

* there are two initials (`W`, `D`) for `WADUDU` ~ `INSECTS`
* need to clarify `WADUDU WENGINE`, which also shares an initial (`D`) and english translation (`INSECTS`) as `WADUDU`
* the initial `W` is associated with both `WADUDU` and `MCHWA`
* different spellings for `SAP`: `UTOMVU` and `UTOMVI`
* `INSECTS` associated with `WADUDU`, `WADUDU WENGINE`, and `SIAFU`
* how should we treat `NA`, `NONE`, and `UNRECORDED`

=== SOLUTION ===

9/4/2026
Ian fixed mifupa, miti and mizizi in FOOD_BOUT by changing to UNRECORDED. DELETED FROM FOOD_PART_LOOKUP
Consolidated insects to 'dudu'
changed all "NA" to "None"
Kept unrecorded
fixed spellings of utomvi and chipukizi

I made all associated changes in FOOD_BOUT, choosing to use names rather than initials

== (#87) There are numerous duplicate fl_sci_food_name_gen values in food_lookup. ==

=== Problem ===

There are 42 instances where fl_sci_food_name_gen values are associated with more than one fl_local_food_name value. This problem assumes that we are using the fl_sci_food_name_gen value for food_lookup.description (i.e., instead of fl_sci_food_name). If instead, food_lookup.description should reflect fl_sci_food_name then this particular problem is moot and can be ignored (but other problems will certainly arise with the switch to fl_sci_food_name).

=== Bad Data ===

<pre>
WITH ranked AS (
SELECT
fl_local_food_name,
fl_sci_food_name,
fl_sci_food_name_gen,
COUNT(*) OVER (
PARTITION BY BTRIM(fl_sci_food_name_gen)
) AS sci_gen_count
FROM clean.food_lookup
WHERE fl_sci_food_name_gen IS NOT NULL
AND BTRIM(fl_sci_food_name_gen) <> ''
)
SELECT
fl_local_food_name,
fl_sci_food_name,
fl_sci_food_name_gen,
sci_gen_count
FROM ranked
WHERE sci_gen_count > 1
ORDER BY fl_sci_food_name_gen, fl_local_food_name, fl_sci_food_name;
</pre>

=== Notes ===

But see problem #85. This problem is only relevant if codes.food_names.description is derived from fl_sci_food_name_gen (i.e., not fl_sci_food_name); else, refer to #85.

=== SOLUTION ===

ALLOW FOR NOW, DISCUSS AS A PI GROUP

==== 2026-09-13 conversion decision ====

Use <code>fl_sci_food_name_gen</code> for
<code>FOOD_NAMES.Description</code>. Exclude lookup rows where
<code>FL_exclude</code> is true before deriving descriptions. Leave each
noncolliding generalized description unchanged. When one generalized
description is shared by multiple eligible local names, derive each target
description as:

<pre>
fl_sci_food_name_gen || ' -- ' || fl_local_food_name
</pre>

This representation is stable and preserves both source values. Do not use
mung's row-order-dependent numeric suffixes. In the 2026-09-13 snapshot, 39
generalized descriptions remained shared by 111 eligible local names after
applying <code>FL_exclude</code>. Rerun a corrected collision query as this
problem is encountered; the historical count of 42 predates the approved
exclusion rule.

== * (#88) There are food_bout rows for which there is not a corresponding food name. ==

=== Problem ===

There are 2,114 clean.food_bout rows for which there is not a matching codes.food_names.name.

=== Bad data ===

==== Full record ====

Exact-match diagnostic: food_bout food names missing from codes.food_names

<pre>
WITH food_rows AS (
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
1 AS seq,
fb_fl_local_food_name AS foodname
FROM clean.food_bout
-- WHERE fb_fl_local_food_name IS NOT NULL
UNION ALL
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
2 AS seq,
fb_local_food_name2 AS foodname
FROM clean.food_bout
-- WHERE NULLIF(fb_local_food_name2, '') IS NOT NULL
)
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
seq,
foodname
FROM food_rows
WHERE foodname NOT IN (
SELECT name
FROM codes.food_names
)
ORDER BY
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
seq,
foodname;
</pre>

==== Summary of non-compliant values ====

Use this to list only the distinct food names present in clean.food_bout but missing from codes.food_names.

<pre>
WITH food_rows AS (
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
1 AS seq,
fb_fl_local_food_name AS raw_foodname,
BTRIM(fb_fl_local_food_name) AS trimmed_foodname
FROM clean.food_bout
-- WHERE fb_fl_local_food_name IS NOT NULL
UNION ALL
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
2 AS seq,
fb_local_food_name2 AS raw_foodname,
BTRIM(fb_local_food_name2) AS trimmed_foodname
FROM clean.food_bout
-- WHERE NULLIF(fb_local_food_name2, '') IS NOT NULL
)
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
seq,
raw_foodname,
trimmed_foodname
FROM food_rows
WHERE raw_foodname NOT IN (
SELECT name
FROM codes.food_names
)
AND trimmed_foodname IN (
SELECT name
FROM codes.food_names
)
ORDER BY
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
seq,
raw_foodname;
</pre>

==== white-space specific mismatches ====

Whitespace-only diagnostic: rows excluded only because of whitespace differences This finds values whose trimmed form exists in codes.food_names, but the raw value does not.

<pre>
WITH food_rows AS (
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
1 AS seq,
fb_fl_local_food_name AS raw_foodname,
BTRIM(fb_fl_local_food_name) AS trimmed_foodname
FROM clean.food_bout
-- WHERE fb_fl_local_food_name IS NOT NULL
UNION ALL
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
2 AS seq,
fb_local_food_name2 AS raw_foodname,
BTRIM(fb_local_food_name2) AS trimmed_foodname
FROM clean.food_bout
-- WHERE NULLIF(fb_local_food_name2, '') IS NOT NULL
)
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
seq,
raw_foodname,
trimmed_foodname
FROM food_rows
WHERE raw_foodname NOT IN (
SELECT name
FROM codes.food_names
)
AND trimmed_foodname IN (
SELECT name
FROM codes.food_names
)
ORDER BY
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
seq,
raw_foodname;
</pre>

=== SOLUTION ===

ALLOW FOR NOW. NEED TO CHECK WHETHER LEGIT NEW FOODS OR JUST POOR HANDWRITING, SPELLING, ETC

==== Karl's comments ====

Ok. Allowing means adding a bunch of odd values, that could continue to be used in the future until the data is cleaned and the "extra" codes removed.

== (#89) There is a food_bout row that has both a compound fb_fpl_local_food_part value and where fb_fpl_local_food_part2 is not null. ==

=== Problem ===

{| class="wikitable"
! FB_FPL_local_food_part !! FB_FPL_local_food_part2
|-
| MATUNDA; CHIPUKIZA || MATUNDA; CHIPUKIZA
|-
|}

=== SOLUTION ===

SHOULD BE ALLOWED

==== 2026-09-13 conversion decision ====

Do not silently choose between a compound primary part and a conflicting
explicit second part. If such a conflict remains after refreshed source
cleanup, exclude the exact source row and document all source columns in the
loader predicate.

The dated 2026-09-13 source contained one conflict:

{| class="wikitable"
! Date !! Focal !! Begin !! End !! Primary part !! Primary name !! Explicit second part !! Second name
|-
| 1985-10-26 || EV || 11:22 || 11:51 || MATUNDA; CHIPUKIZI || MBULA || MATUNDA; CHIPUKIZI || BISHURUSHURU
|}

Under the approved parser, the compound second token is
<code>CHIPUKIZI</code>, while the explicit second value is the entire compound
<code>MATUNDA; CHIPUKIZI</code>. The older table above uses the stale spelling
<code>CHIPUKIZA</code>. Reproduce the complete current row exactly before
adding the exclusion; do not rely on this dated spelling or count.

== * (#90) There are food_bout rows for which the fb_fpl_local_food_part or fb_fpl_local_food_part2 does not have a matching value in codes.food_parts. ==

=== Problem ===

There are 229 records for which a food_bout row does not have a match between for which the fb_fpl_local_food_part or fb_fpl_local_food_part2 and codes.food_parts. Note, however, that the record count is somewhat ambiguous since we are deailing with multiple parts (1 and 2), and multiple rows per event.

=== Bad data ===

==== Full record ====

diagnostic: fb_fpl_local_food_part or fb_fpl_local_food_part2 does not have a matching value in codes.food_parts

<pre>
WITH prepared AS (
SELECT
fb.*,
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
) AS part1_norm,
UPPER(
NULLIF(
BTRIM(
split_part(
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
),
';',
1
)
),
''
)
) AS seq1_part,
CASE
WHEN regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
) LIKE '%;%'
THEN UPPER(
NULLIF(
BTRIM(
split_part(
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
),
';',
2
)
),
''
)
)
ELSE UPPER(NULLIF(BTRIM(fb.fb_fpl_local_food_part2), ''))
END AS seq2_part
FROM clean.food_bout fb
)
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
fb_fpl_local_food_part,
fb_fpl_local_food_part2,
seq1_part,
seq2_part,
CASE
WHEN seq1_part NOT IN (SELECT part FROM codes.food_parts)
THEN 'seq1_part missing from codes.food_parts'
WHEN seq2_part IS NOT NULL
AND seq2_part NOT IN (SELECT part FROM codes.food_parts)
THEN 'seq2_part missing from codes.food_parts'
END AS exclusion_reason
FROM prepared
WHERE seq1_part NOT IN (SELECT part FROM codes.food_parts)
OR (
seq2_part IS NOT NULL
AND seq2_part NOT IN (SELECT part FROM codes.food_parts)
)
ORDER BY
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time;
</pre>

==== Summary of non-compliant values ====

<pre>
WITH prepared AS (
SELECT
UPPER(
NULLIF(
BTRIM(
split_part(
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
),
';',
1
)
),
''
)
) AS seq1_part,
CASE
WHEN regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
) LIKE '%;%'
THEN UPPER(
NULLIF(
BTRIM(
split_part(
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
),
';',
2
)
),
''
)
)
ELSE UPPER(NULLIF(BTRIM(fb.fb_fpl_local_food_part2), ''))
END AS seq2_part
FROM clean.food_bout fb
)
SELECT missing_part, COUNT(*) AS occurrences
FROM (
SELECT seq1_part AS missing_part
FROM prepared
WHERE seq1_part NOT IN (SELECT part FROM codes.food_parts)

UNION ALL

SELECT seq2_part AS missing_part
FROM prepared
WHERE seq2_part IS NOT NULL
AND seq2_part NOT IN (SELECT part FROM codes.food_parts)
) x
GROUP BY missing_part
ORDER BY missing_part;
</pre>

=== SOLUTION ===

ALLOW FOR NOW. NEED TO HAND CHECK AND DECIDE ON PROTOCOL

==== Karl's comments ====

Stevan can chime in here, but I think this means adding more illegitimate "legitimate" codes.

== * (#91) There are food_bout rows that have fb_begin_feed_time > fb_end_feed_time. ==

=== Problem ===

There are 43 clean.food_bout rows that have fb_begin_feed_time > fb_end_feed_time.

=== Bad data ===

<pre>
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
fb_begin_feed_time - fb_end_feed_time AS inverted_by,
fb_fpl_local_food_part,
fb_fl_local_food_name
FROM clean.food_bout
WHERE fb_begin_feed_time IS NOT NULL
AND fb_end_feed_time IS NOT NULL
AND fb_begin_feed_time > fb_end_feed_time
ORDER BY fb_fol_date, fb_fol_b_focal_animid, fb_begin_feed_time;
</pre>

=== SOLUTION ===
IAN/VOLUNTEER TO FIX

== * (#92) There are food_bout rows that do not have a corresponding follow. ==

=== Problem ===

clean.food_bout references follows by (fb_fol_b_focal_animid, fb_fol_date), but not all rows (n=455) match sokwedb.follows exactly.

=== Bad data ===

Includes both:
* Normalization-only matches: match after TRIM+UPPER (raw focal text quality issue).
* True misses: no matching follow even after normalization (actual source data gap).

<pre>
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
fb_begin_feed_time - fb_end_feed_time AS inverted_by,
fb_fpl_local_food_part,
fb_fl_local_food_name
FROM clean.food_bout
WHERE fb_begin_feed_time IS NOT NULL
AND fb_end_feed_time IS NOT NULL
AND fb_begin_feed_time > fb_end_feed_time
ORDER BY fb_fol_date, fb_fol_b_focal_animid, fb_begin_feed_time;
</pre>

=== SOLUTION ===
NEED TO LOOK AT THE SOURCE OF THE PROBLEM B/C NO-ONE ENTERS FEEDING FROM B-REC NOTES.

==== Karl's comments ====

I think it makes sense to normalize the data in the clean schema, upper casing and removing spaces. And then deal with what's left.

== (#93) Need to address how we treat behaviours for the second value of food parts AND how they are derived. ==

=== Problem ===

1. In load_food_events.sql, we are loading into table food_events, where each row requires both:

* foodpart (NOT NULL + FK)
* foodname (NOT NULL + FK)

...but for seq=2, these cases behave differently:

# part2 exists, name2 missing: insert NODATA to name2
# part2 missing, name2 exists: insert NODATA to part2
# both missing: no seq=2 row.
# both exist: normal seq=2 row.

This approach required adding NODATA to codes.food_names and codes.food_parts.

Require approval for this approach or an altnernative.

2. Multiple food parts can be dervied in one of two ways:

# there are values for both fb_fpl_local_food_part and fb_fpl_local_food_part2, or
# there are multiple food parts in fb_fpl_local_food_part separated by a `;` (usually) or, rarely, `:`

Do we need to document or log if fb_fpl_local_food_part2 is derived from a compound fb_fpl_local_food_part value?

=== Bad data ===

==== count of values that meet these conditions ====

<pre>
WITH prepared AS (
SELECT
fb.*,
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
) AS part1_norm,
UPPER(
NULLIF(
BTRIM(
split_part(
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
),
';',
2
)
),
''
)
) AS seq2_from_part1,
UPPER(NULLIF(BTRIM(fb.fb_fpl_local_food_part2), '')) AS seq2_from_part2,
NULLIF(BTRIM(fb.fb_local_food_name2), '') AS seq2_name
FROM clean.food_bout fb
),
effective AS (
SELECT
*,
CASE
WHEN part1_norm LIKE '%;%' THEN seq2_from_part1
ELSE seq2_from_part2
END AS seq2_part
FROM prepared
),
seq2_rows AS (
SELECT
COALESCE(seq2_part, 'NODATA') AS final_seq2_part,
COALESCE(seq2_name, 'NODATA') AS final_seq2_name
FROM effective
WHERE seq2_part IS NOT NULL
OR seq2_name IS NOT NULL
)
SELECT
CASE
WHEN final_seq2_part = 'NODATA' AND final_seq2_name = 'NODATA' THEN 'both_nodata'
WHEN final_seq2_part = 'NODATA' THEN 'part_nodata_only'
WHEN final_seq2_name = 'NODATA' THEN 'name_nodata_only'
ELSE 'no_nodata'
END AS seq2_case,
COUNT(*) AS rows
FROM seq2_rows
GROUP BY 1
ORDER BY 1;
</pre>

==== full record ====

<pre>
WITH prepared AS (
SELECT
fb.*,
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
) AS part1_norm,
UPPER(
NULLIF(
BTRIM(
split_part(
regexp_replace(
COALESCE(fb.fb_fpl_local_food_part, ''),
E'\\s*[:;]\\s*',
';',
'g'
),
';',
2
)
),
''
)
) AS seq2_from_part1,
UPPER(NULLIF(BTRIM(fb.fb_fpl_local_food_part2), '')) AS seq2_from_part2,
NULLIF(BTRIM(fb.fb_local_food_name2), '') AS seq2_name
FROM clean.food_bout fb
),
effective AS (
SELECT
*,
CASE
WHEN part1_norm LIKE '%;%' THEN seq2_from_part1
ELSE seq2_from_part2
END AS seq2_part
FROM prepared
)
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_fpl_local_food_part,
fb_fpl_local_food_part2,
fb_fl_local_food_name,
fb_local_food_name2,
COALESCE(seq2_part, 'NODATA') AS final_seq2_part,
COALESCE(seq2_name, 'NODATA') AS final_seq2_name
FROM effective
WHERE seq2_part IS NOT NULL
OR seq2_name IS NOT NULL
ORDER BY
fb_fol_date,
fb_fol_b_focal_animid,
fb_fl_local_food_name;
</pre>

=== SOLUTION ===
GET STEVAN TO EXPLAIN TO ICG AND ELO

==== 2026-09-13 conversion decision ====

Add <code>NODATA</code>, described as <code>No data recorded</code>, to both
<code>FOOD_NAMES</code> and <code>FOOD_PARTS</code>. Create a Seq 2 row when
either approved secondary component exists. Use <code>NODATA</code> only for
the missing half of that pair:

* part exists, name missing: use <code>NODATA</code> for FoodName;
* name exists, part missing: use <code>NODATA</code> for FoodPart;
* neither exists: create no Seq 2 row; and
* both exist: preserve both approved values.

Normalize colon and semicolon delimiters and derive ordered compound tokens for
Seq 1 and Seq 2. If a compound-derived second part conflicts with an explicit
second part, exactly exclude the row under Problem #89. Do not silently apply
precedence. The dated source had 12 secondary names without an explicit second
part; these are eligible for the approved <code>NODATA</code> FoodPart rather
than exclusion. Rerun both queries above with the approved clean normalization
and exclusion projection as this problem is encountered.

== * (#94) There are food_bout rows where the start or end of the observation is outside of allowable bounds. ==

=== Problem ===

There are 11 record where the food_bout is is outside of allowable bounds.

=== Bad data ===

<pre>
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
CASE
WHEN fb_begin_feed_time IS NULL THEN 'BEGIN_NULL'
WHEN fb_begin_feed_time < '04:00:00'::time THEN 'BEGIN_BEFORE_MIN'
WHEN fb_begin_feed_time > '20:00:00'::time THEN 'BEGIN_AFTER_MAX'
ELSE 'BEGIN_OK'
END AS begin_status,
CASE
WHEN fb_end_feed_time IS NULL THEN 'END_NULL'
WHEN fb_end_feed_time < '04:00:00'::time THEN 'END_BEFORE_MIN'
WHEN fb_end_feed_time > '20:00:00'::time THEN 'END_AFTER_MAX'
ELSE 'END_OK'
END AS end_status,
COALESCE(
LEAST(GREATEST(fb_begin_feed_time, '04:00:00'::time), '20:00:00'::time),
'04:00:00'::time
) AS clamped_begin,
COALESCE(
LEAST(GREATEST(fb_end_feed_time, '04:00:00'::time), '20:00:00'::time),
'04:00:00'::time
) AS clamped_end
FROM clean.food_bout
WHERE fb_begin_feed_time IS NULL
OR fb_end_feed_time IS NULL
OR fb_begin_feed_time < '04:00:00'::time
OR fb_begin_feed_time > '20:00:00'::time
OR fb_end_feed_time < '04:00:00'::time
OR fb_end_feed_time > '20:00:00'::time
ORDER BY fb_fol_date, fb_fol_b_focal_animid, fb_begin_feed_time;
</pre>

=== SOLUTION ===
ALLOW FOR NOW. IAN TO FIX

==== Karl's comment ====

See remarks on problem #81.

== (#95) There is at least one food_bout rows where the start or stop time has a seconds value. ==

=== Problem ===

There is one food_bout row where the start or stop time has a seconds value, which violates our time-check constraints.

=== Bad data ===

<pre>
SELECT
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time
FROM clean.food_bout
WHERE EXTRACT(SECOND FROM fb_begin_feed_time) <> 0
OR EXTRACT(SECOND FROM fb_end_feed_time) <> 0
ORDER BY fb_fol_date, fb_fol_b_focal_animid, fb_begin_feed_time;
</pre>

=== SOLUTION ===

IAN (HOPEFULLY) FIXED IN ACCESS 7/3

==== Karl's comment ====

Stevan, this is supposed to be fixed in the transition from the <code>raw</code> schema to the <code>tidy</code> schema. At least this is where the date and time intervals are normalized.

Maybe it got missed, or just needed bringing up as an issue. No real point in fixing it in MS Access. Doing so feels like losing information, and possibly being inconsistent in the way time intervals are normalized.

Update 2026-07-22 SRE cannot reproduce this error, e.g., see below querying the tidy schema, which returns zero records; calling this resolved.

<pre>
SELECT
"FB_FOL_date",
"FB_FOL_B_focal_AnimID",
"FB_begin_feed_time",
"FB_end_feed_time"
FROM tidy."FOOD_BOUT"
WHERE EXTRACT(SECOND FROM "FB_begin_feed_time") <> 0
OR EXTRACT(SECOND FROM "FB_end_feed_time") <> 0
ORDER BY "FB_FOL_date", "FB_FOL_B_focal_AnimID", "FB_begin_feed_time";
</pre>

== (#96) There are GROOMINGS rows that have direction values that do not match to mappable values. ==

=== Problem ===

There are 26 GROOMINGS rows that have have a direction other than 'G', 'R', 'M', 'U', which are the only values that map to an allowable direction.

=== Bad data ===

==== Summary of non-compliant values ====

<pre>
SELECT
COALESCE(grm_direction, '<NULL>') AS raw_direction,
quote_literal(grm_direction) AS quoted_direction,
UPPER(COALESCE(grm_direction, '')) AS upper_no_trim,
LENGTH(COALESCE(grm_direction, '')) AS raw_length,
COUNT(*) AS row_count
FROM clean.groom_bout
WHERE UPPER(COALESCE(grm_direction, '')) NOT IN ('G', 'R', 'M', 'U')
GROUP BY
COALESCE(grm_direction, '<NULL>'),
quote_literal(grm_direction),
UPPER(COALESCE(grm_direction, '')),
LENGTH(COALESCE(grm_direction, ''))
ORDER BY row_count DESC, raw_direction;
</pre>

==== Full record ====

<pre>
SELECT
grm_fol_date,
grm_fol_b_focal_animid,
grm_b_partner_animid,
grm_direction,
quote_literal(grm_direction) AS quoted_direction,
UPPER(COALESCE(grm_direction, '')) AS upper_no_trim,
LENGTH(COALESCE(grm_direction, '')) AS raw_length,
(
COALESCE(grm_direction, '') ~ '^[[:space:]]'
OR COALESCE(grm_direction, '') ~ '[[:space:]]$'
) AS has_edge_space,
grm_problems,
grm_comments,
grm_extracted_by
FROM clean.groom_bout
WHERE UPPER(COALESCE(grm_direction, '')) NOT IN ('G', 'R', 'M', 'U')
ORDER BY grm_fol_date, grm_fol_b_focal_animid, grm_b_partner_animid;
</pre>

====SOLUTION====
ICG FIXED IN ACCESS 8/18/2026

== (#97) There are GROOMINGS rows that have certainty values that do not match to mappable values. ==

=== Problem ===

There are 303 GROOMINGS rows that have have a direction other than 'Y' or 'N', which are the only values that map to an allowable certainty (Y -> 1; N -> 0). Only values of 0 and 1 are allowed in events.certainty.

=== Bad data ===

==== Summary of non-compliant values ====

<pre>
SELECT
COALESCE(grm_time_certainty, '<NULL>') AS raw_certainty,
quote_literal(grm_time_certainty) AS quoted_certainty,
UPPER(COALESCE(grm_time_certainty, '')) AS upper_no_trim,
LENGTH(COALESCE(grm_time_certainty, '')) AS raw_length,
COUNT(*) AS row_count
FROM clean.groom_bout
WHERE UPPER(COALESCE(grm_time_certainty, '')) NOT IN ('Y', 'N')
GROUP BY
COALESCE(grm_time_certainty, '<NULL>'),
quote_literal(grm_time_certainty),
UPPER(COALESCE(grm_time_certainty, '')),
LENGTH(COALESCE(grm_time_certainty, ''))
ORDER BY row_count DESC, raw_certainty;
</pre>

==== Full record ====

<pre>
SELECT
grm_fol_date,
grm_fol_b_focal_animid,
grm_b_partner_animid,
grm_time_certainty,
quote_literal(grm_time_certainty) AS quoted_certainty,
UPPER(COALESCE(grm_time_certainty, '')) AS upper_no_trim,
LENGTH(COALESCE(grm_time_certainty, '')) AS raw_length,
(
COALESCE(grm_time_certainty, '') ~ '^[[:space:]]'
OR COALESCE(grm_time_certainty, '') ~ '[[:space:]]$'
) AS has_edge_space,
grm_problems,
grm_comments,
grm_extracted_by
FROM clean.groom_bout
WHERE UPPER(COALESCE(grm_time_certainty, '')) NOT IN ('Y', 'N')
ORDER BY grm_fol_date, grm_fol_b_focal_animid, grm_b_partner_animid;
</pre>

=== SOLUTION ===

ICG FIXED IN ACCESS 8/18/2026

== (#98) --- empty placeholder to restore count ==

== (#99) There are GROOMINGS rows for which the extractedBy values is empty. ==

=== Problem ===

There are 10,710 GROOMINGS rows for which the extractedBy values is empty. This value cannot be NULL.

=== Bad data ===

<pre>
SELECT
grm_fol_date,
grm_fol_b_focal_animid,
grm_b_partner_animid,
grm_direction,
grm_time_certainty,
grm_extracted_by,
quote_literal(grm_extracted_by) AS quoted_extracted_by,
LENGTH(COALESCE(grm_extracted_by, '')) AS raw_length,
grm_problems,
grm_comments
FROM clean.groom_bout
WHERE NULLIF(BTRIM(grm_extracted_by), '') IS NULL
ORDER BY grm_fol_date, grm_fol_b_focal_animid, grm_b_partner_animid;
</pre>
===SOLUTION===
ICG CHANGED ALL NULLS TO 'UNK' IN ACCESS. 8/18/2026

== * (#100) There are GROOMINGS rows that have extractedby values that do not match to mappable people values. ==

=== Problem ===

There are 14,771 GROOMINGS rows that have extractedby values that do not match to mappable codes.people values.

=== Bad data ===

==== Summary of non-compliant values ====

<pre>
SELECT
BTRIM(gb.grm_extracted_by) AS normalized_extracted_by,
COUNT(*) AS row_count
FROM clean.groom_bout gb
WHERE NULLIF(BTRIM(gb.grm_extracted_by), '') IS NOT NULL
AND NOT EXISTS (
SELECT 1
FROM codes.people p
WHERE p.person = BTRIM(gb.grm_extracted_by)
)
GROUP BY BTRIM(gb.grm_extracted_by)
ORDER BY row_count DESC, normalized_extracted_by;
</pre>

==== Full record ====

<pre>
SELECT
gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_b_partner_animid,
gb.grm_extracted_by,
BTRIM(gb.grm_extracted_by) AS normalized_extracted_by,
gb.grm_direction,
gb.grm_time_certainty,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout gb
WHERE NULLIF(BTRIM(gb.grm_extracted_by), '') IS NOT NULL
AND NOT EXISTS (
SELECT 1
FROM codes.people p
WHERE p.person = BTRIM(gb.grm_extracted_by)
)
ORDER BY gb.grm_fol_date, gb.grm_fol_b_focal_animid, gb.grm_b_partner_animid;
</pre>

== * (#101) There are AGGRESSION_EVENT rows that have NULL focals, a NULL ae_fol_b_focal_id value ==

=== Problem ===

There are 144 aggression_event rows that have NULL for the focal.

=== Bad data ===

<pre>
SELECT *
FROM easy.aggression_event
WHERE ae_fol_b_focal_id IS NULL;
</pre>

=== Solution ===

Change these values to the unknown individual, <code>UNK</code>.
This means that these rows are converted to WATCHES rows with a Type of <code>B</code>, and <code>UNK</code> as the focal.

The data needs to be reviewed to see if this is appropriate. See also problem #102.

== * (#102) There are AGGRESSION_EVENT rows that have NULL times, a NULL ae_time value ==

=== Problem ===

There are 107 aggression_event rows that have NULL for the time.
Note that all of these rows also have NULL for the focal, problem #101.

=== Bad data ===

<pre>
SELECT *
FROM easy.aggression_event
WHERE ae_time IS NULL;
</pre>

=== Solution ===

Change these values to the unknown time, midnight, <code>00:00</code>.
These need to be reviewed to ensure the change is appropriate.
Meeting discussions indicate the many, ideally all, of these rows are the result of perusal of a book's content, where aggressions were mentioned but these aggressions did not otherwise appear in the data.

== (#103) There are BIOGRAPHY rows that have empty birthgroup values. ==

=== Problem ===

There are 64 BIOGRAPHY rows that have empty strings for birthgroup; NULL values are allowed but not empty strings. Some of the offending records are addressed also in Problem #13.

=== Bad data ===

''note: (temporary) solution applied to clean so query tidy''

<pre>
SELECT "B_AnimID",
"B_AnimName",
quote_nullable("B_BirthGroup") AS birthgroup,
length("B_BirthGroup") AS character_length,
octet_length("B_BirthGroup") AS byte_length,
"B_BGCertainty"
FROM tidy."BIOGRAPHY"
WHERE "B_BirthGroup" IS NOT NULL
AND BTRIM("B_BirthGroup") = ''
ORDER BY "B_AnimID";
</pre>

=== Solution ===

Propose converting empty strings to NULL in the clean schema, PI to confirm.

<pre>
UPDATE biography
SET b_birthgroup = NULLIF(BTRIM(b_birthgroup), '')
WHERE b_birthgroup IS DISTINCT FROM NULLIF(BTRIM(b_birthgroup), '');
</pre>

Addressed with commit f2ddb85cd22d3e5233d049d92ae8cb361a538114

== (#104) There are BIOGRAPHY rows that have empty momid values. ==

=== Problem ===

There are 89 BIOGRAPHY rows that have empty strings for momid; NULL values are allowed but not empty strings.

=== Bad data ===

''note: (temporary) solution applied to clean so query tidy''

<pre>
SELECT "B_AnimID",
"B_AnimName",
quote_nullable("B_MomID") AS momid,
length("B_MomID") AS character_length,
octet_length("B_MomID") AS byte_length,
"B_Birthdate"
FROM tidy."BIOGRAPHY"
WHERE "B_MomID" IS NOT NULL
AND BTRIM("B_MomID") = ''
ORDER BY "B_AnimID";
</pre>

=== Solution ===

Propose converting empty strings to NULL in the clean schema, PI to confirm.

<pre>
UPDATE biography
SET b_momid = NULLIF(BTRIM(b_momid), '')
WHERE b_momid IS DISTINCT FROM NULLIF(BTRIM(b_momid), '');
</pre>

Addressed with commit 5e0e787825e847eac1cc93375e26ea1d3f715d3e

== (#105) There is a comm member log record that is missing a description. ==

=== Problem ===

There is a single comm member log records where the description is missing.

=== Bad data ===

''note: querying tidy''

<pre>
SELECT date_of_update,
chimp_id,
quote_nullable(update_description) AS update_description,
length(update_description) AS description_length,
update_rationale,
"made by"
FROM tidy."COMMUNITY_MEMBERSHIP_UPDATE_LOG"
WHERE update_description IS NULL
OR BTRIM(update_description) = ''
ORDER BY date_of_update,
chimp_id,
update_rationale,
"made by";
</pre>

===solution===

Fixed in Access by ICG - duplicated update_rationale

== (#106) There are GROOM_SCANS records whose extracted-by person is not in PEOPLE. ==

=== Problem ===

There are 44,679 GROOM_SCANS records where the GS_extracted_by value is not a PEOPLE.Person value. All GROOM_SCANS records are affected. There are four distinct offending values: KASEN, KAREN MCLELLAN, Anika Richter, and Janelle Carmichael.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
quote_nullable(gs.gs_extracted_by) AS gs_extracted_by,
length(gs.gs_extracted_by) AS extracted_by_length
FROM clean.groom_scans AS gs
WHERE NOT EXISTS (
SELECT 1
FROM codes.people AS people
WHERE people.person = gs.gs_extracted_by)
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
</pre>

=== Solution ===

During construction of the clean schema, add one deterministic spelling of each case-insensitive GROOM_SCANS extractor value to clean.people when no case-insensitive PEOPLE match already exists. Prefer an existing PEOPLE spelling, and otherwise prefer a mixed-case source spelling over an all-uppercase or all-lowercase spelling. Rewrite gs_extracted_by to the exact spelling stored in clean.people.

The ordinary support-table loader copies these rows to codes.people with active set to true before B-record groom scans are loaded. This preserves all 44,679 source rows and leaves the production foreign key and active-person rule intact.

== * (#107) There are GROOM_SCAN_AREC records having the same chimpanzee as both grooming participants. ==

=== Problem ===

There are 1,784 GROOM_SCAN_AREC records where Chimp_1 and Chimp_2 are the same individual. The records span 210 dates and 39 distinct chimpanzees, and all have a Direction value of M. The production loader converts Direction M into two Mutual ROLES rows. These records cannot be loaded because a participant may occur only once in an event and the attendance groom-scan rules require two different participants. The project investigators must determine whether these records represent self-grooming or erroneous participant data and how they should be represented in the production database.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
gs.direction
FROM clean.groom_scan_arec AS gs
WHERE gs.chimp_1 = gs.chimp_2
ORDER BY gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2;
</pre>

== * (#108) There are GROOM_SCAN_AREC participants that are not in BIOGRAPHY_DATA. ==

=== Problem ===

There are 1,124 GROOM_SCAN_AREC records having at least one participant that is not a BIOGRAPHY_DATA.AnimID value. There are 81 distinct missing participant values. The missing value occurs in Chimp_1 in 112 records and in Chimp_2 in 1,016 records; four records have missing values in both columns. None of the 81 values are present in clean.BIOGRAPHY, from which BIOGRAPHY_DATA is loaded. The production loader cannot create the related ROLES rows until the project investigators determine whether these values identify individuals that should be added to BIOGRAPHY, corrected to existing AnimID values, or otherwise represented.

=== Bad data ===

''note: querying clean, the schema read by the production loader, and sokwedb.BIOGRAPHY_DATA, the referenced production table''

<pre>
SELECT gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
gs.direction,
bad_participant.source_column,
quote_nullable(bad_participant.animid) AS missing_animid,
length(bad_participant.animid) AS animid_length
FROM clean.groom_scan_arec AS gs
CROSS JOIN LATERAL (
VALUES ('Chimp_1', gs.chimp_1),
('Chimp_2', gs.chimp_2)
) AS bad_participant(source_column, animid)
WHERE NOT EXISTS (
SELECT 1
FROM sokwedb.biography_data AS bio
WHERE bio.animid = bad_participant.animid)
ORDER BY gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
bad_participant.source_column;
</pre>

===SOLUTION===

HAVE A STUDENT CHECK AGAINST PAPER

== * (#109) There are GROOM_SCAN_AREC records whose dates have no ATTENDANCE records from which to obtain a community. ==

=== Problem ===

There are 5,111 GROOM_SCAN_AREC records on 134 dates for which there is no ATTENDANCE record on the same date. GROOM_SCAN_AREC does not contain a community value, and the production loader obtains the WATCHES.CommID value from ATTENDANCE.A_CL_Community_ID by date. Consequently, the loader cannot create the required WATCHES row for these scans without assuming a community. After the temporary exclusions for Problems #107 and #108, 4,786 otherwise-loadable records on 133 dates are affected. The project investigators must determine the appropriate community for these scan dates.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
gs.direction
FROM clean.groom_scan_arec AS gs
WHERE NOT EXISTS (
SELECT 1
FROM clean.attendance AS attendance
WHERE attendance.a_date = gs.date)
ORDER BY gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2;
</pre>

== * (#110) There are GROOM_SCAN_AREC records involving individuals after their departure from the study. ==

=== Problem ===

There are 2,090 GROOM_SCAN_AREC records where at least one participant's scan date is after the individual's BIOGRAPHY_DATA.DepartDate. The records involve 18 distinct individuals on 177 dates. The offending individual occurs in Chimp_1 in 1,928 records and in Chimp_2 in 162 records; no record has both participants after their departure dates. After the temporary exclusions for Problems #107, #108, and #109, 1,725 otherwise-loadable records on 147 dates are affected. The production schema requires event participants to be under study on the WATCHES.Date, so these records cannot be loaded until the project investigators determine whether the scan participants or biography departure dates should be corrected.

The trigger error originally reported PT's EntryDate, 1970-09-11, as its DepartDate due to an error in the trigger's diagnostic query. PT's actual BIOGRAPHY_DATA.DepartDate is 1973-04-17, which is before the first affected PT scan on 1973-05-25. The constraint comparison itself used the correct DepartDate and therefore rejected the row correctly.

=== Bad data ===

''note: querying clean, the schema read by the production loader, and sokwedb.BIOGRAPHY_DATA, the referenced production table''

<pre>
SELECT gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
gs.direction,
bad_participant.source_column,
bad_participant.animid,
bio.departdate,
gs.date - bio.departdate AS days_after_departure
FROM clean.groom_scan_arec AS gs
CROSS JOIN LATERAL (
VALUES ('Chimp_1', gs.chimp_1),
('Chimp_2', gs.chimp_2)
) AS bad_participant(source_column, animid)
JOIN sokwedb.biography_data AS bio
ON bio.animid = bad_participant.animid
WHERE gs.date > bio.departdate
ORDER BY gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
bad_participant.source_column;
</pre>

== * (#111) There are GROOM_SCAN_AREC records involving individuals before their entry into the study. ==

=== Problem ===

There are 130 GROOM_SCAN_AREC records where at least one participant's scan date is before the individual's BIOGRAPHY_DATA.EntryDate. The records involve 10 distinct individuals on 21 dates. The offending individual occurs in Chimp_1 in 37 records and in Chimp_2 in 93 records; no record has both participants before their entry dates. After the temporary exclusions for Problems #107 through #110, 30 otherwise-loadable records on 11 dates are affected. The production schema requires event participants to be under study on the WATCHES.Date, so these records cannot be loaded until the project investigators determine whether the scan participants or biography entry dates should be corrected.

=== Bad data ===

''note: querying clean, the schema read by the production loader, and sokwedb.BIOGRAPHY_DATA, the referenced production table''

<pre>
SELECT gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
gs.direction,
bad_participant.source_column,
bad_participant.animid,
bio.entrydate,
bio.entrydate - gs.date AS days_before_entry
FROM clean.groom_scan_arec AS gs
CROSS JOIN LATERAL (
VALUES ('Chimp_1', gs.chimp_1),
('Chimp_2', gs.chimp_2)
) AS bad_participant(source_column, animid)
JOIN sokwedb.biography_data AS bio
ON bio.animid = bad_participant.animid
WHERE gs.date < bio.entrydate
ORDER BY gs.date,
gs.time,
gs.chimp_1,
gs.chimp_2,
bad_participant.source_column;
</pre>

== (#112) GROOM_BOUT uses U as an unknown initiator or terminator animal ID ==

=== Problem ===

Some GROOM_BOUT rows contain U in GRM_B_initiator_AnimId or GRM_B_terminator_AnimId. Although U is a valid value for GRM_direction, it is not a valid animal ID.

The production GROOMINGS.Initiator and GROOMINGS.Terminator columns reference participant rows in ROLES. During conversion, the loader creates roles for the focal and partner animals and then looks up the role corresponding to the recorded initiator or terminator. A value of U cannot match either participant, causing the loader to fail with query returned no rows.

This problem became visible after the Problem #99 and Problem #100 exclusions were removed. Those exclusions had prevented many affected grooming bouts from reaching the initiator and terminator lookup.

=== Bad data ===

The following query identifies every occurrence of `U` in the initiator and terminator animal-ID fields:

<pre>
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
gb.grm_direction,
invalid_participant.source_column,
invalid_participant.animid AS offending_animid,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
CROSS JOIN LATERAL (
VALUES ('GRM_B_initiator_AnimId', gb.grm_b_initiator_animid),
('GRM_B_terminator_AnimId', gb.grm_b_terminator_animid)
) AS invalid_participant(source_column, animid)
WHERE BTRIM(invalid_participant.animid) = 'U'
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
invalid_participant.source_column;
</pre>

=== Solution ===

During construction of the clean schema, convert U in groom_bout.grm_b_initiator_animid and groom_bout.grm_b_terminator_animid to SQL NULL. Do not change groom_bout.grm_direction when it contains U, because that is a valid direction code representing unknown grooming direction.

The grooming loader already treats a NULL initiator or terminator as unknown and stores NULL in the corresponding production column without attempting to find a participant role.

Other nonempty initiator or terminator animal IDs that do not identify either the focal or partner animal should remain conversion errors and be reviewed separately rather than being converted automatically to NULL.

Approved by the investigator on 2026-09-11.

== (#113) The Access BIOGRAPHY.NONE row lacks values required by BIOGRAPHY_DATA ==

=== Problem ===

The A-record groom-scan conversion represents feeding-station scans as A-record WATCHES. These scans have no focal individual, so the conversion uses the special AnimID <code>NONE</code> for the related WATCHES row rather than incorrectly identifying the focal individual as unknown with <code>UNK</code>. The production BIOGRAPHY_DATA table must therefore contain a <code>NONE</code> row.

The conversion originally created this row in <code>clean.BIOGRAPHY</code>. After the row was added to the Access BIOGRAPHY table, retaining that INSERT caused the clean-schema build to fail with a duplicate <code>BIOGRAPHY</code> primary key for <code>NONE</code>. Removing the INSERT exposed a second problem: the Access row contains NULL in fields that are required by BIOGRAPHY_DATA, beginning with <code>BCCertainty</code>, so it cannot be loaded directly into the production table.

=== Bad data ===

The affected row is the single Access BIOGRAPHY record whose <code>B_AnimID</code> is <code>NONE</code>. A diagnostic query is unnecessary because this is an intentionally defined sentinel record rather than an unidentified set of source-data rows.

=== Solution ===

Keep Access authoritative for the existence of the <code>NONE</code> BIOGRAPHY row. During construction of the <code>clean</code> schema, update only that row with the previously established no-focal placeholder values required by BIOGRAPHY_DATA. Do not insert a second row, modify the restored Access data in <code>raw</code>, or apply the correction directly to <code>sokwedb</code>.

This preserves the decision that A-record groom scans use <code>NONE</code> when no focal individual exists while allowing the Access-supplied sentinel record to satisfy the production biography constraints.

== (#114) BRECORD_NOTES times are not uniformly compatible with EVENTS ==

=== Problem ===

The Access BRECORD_NOTES.BREC_time column is stored as a timestamp and is converted to a PostgreSQL TIME value in the tidy schema. Problem #10 corrected records whose timestamp had the wrong date component, but converting the datatype preserves seconds. The production EVENTS.Start and EVENTS.Stop columns record times to the minute and reject nonzero seconds.

Some B-record notes also use midnight to indicate that no time was recorded. The production schema previously reserved the sdb_no_time midnight sentinel for aggression events, so it rejected otherwise-loadable B-record notes with no recorded time.

In the local conversion data examined on 2026-08-20, 22 rows contained nonzero seconds, 46 rows contained midnight, and no rows had a SQL NULL time. Forty-five of the midnight rows had a matching follow-derived B-record watch.

=== Bad data ===

<pre>
SELECT brec_fol_date,
brec_fol_b_focal_animid,
brec_time,
extract(second FROM brec_time) AS seconds
FROM clean.brecord_notes
WHERE brec_time IS NULL
OR brec_time = '00:00'::TIME
OR extract(second FROM brec_time) <> 0
ORDER BY brec_fol_date,
brec_fol_b_focal_animid,
brec_time;
</pre>

=== Solution ===

Truncate nonzero seconds from BREC_time in tidy_cleanups.sql before the column is converted to TIME. Do not round the values. Convert a SQL NULL time to sdb_no_time at the production load boundary and preserve an existing midnight value as that sentinel. Permit both aggression and B-record-note EVENTS rows to use sdb_no_time, with both Start and Stop set to the sentinel.

== * (#115) BRECORD_NOTES rows lack a matching follow-derived WATCHES row ==

=== Problem ===

A production BRECORD_NOTES row belongs to an EVENTS row, which in turn must belong to a WATCHES row. The ordinary B-record watches are created from clean.follow. Some B-record notes have no follow with the same focal individual and date. It is not yet known which records truly lack a follow and which contain an incorrect focal or date.

Creating provisional watches would require guessing the focal, community, or date and would make the affected records difficult to identify for later review. In the local conversion data examined on 2026-08-20, 35,517 of 603,977 B-record-note rows lacked a matching follow.

=== Bad data ===

<pre>
SELECT brecord_notes.*
FROM clean.brecord_notes
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow
WHERE follow.fol_b_animid =
brecord_notes.brec_fol_b_focal_animid
AND follow.fol_date = brecord_notes.brec_fol_date)
ORDER BY brec_fol_date,
brec_fol_b_focal_animid,
brec_time;
</pre>

=== Solution ===

Temporarily exclude these rows from the production conversion. Retain them unchanged in clean.brecord_notes so that the focal and date can be reviewed and corrected in Access. The loader must reuse only an existing type-B WATCHES row having the same focal and date; it must not create a watch for an unmatched B-record note.

== * (#116) BRECORD_NOTES contains times outside the EVENTS limits ==

=== Problem ===

Except for the sdb_no_time sentinel, production event times must be between sdb_min_event_start (04:00) and sdb_max_event_stop (20:00), inclusive. Some B-record-note times are outside those limits and their correct values must be determined from the source records.

In the local conversion data examined on 2026-08-20, 751 rows had an out-of-range time other than midnight. Of those, 711 had a matching follow-derived watch and would otherwise be loadable.

=== Bad data ===

<pre>
SELECT brecord_notes.*
FROM clean.brecord_notes
WHERE brec_time IS NOT NULL
AND brec_time <> '00:00'::TIME
AND (brec_time < '04:00'::TIME
OR brec_time > '20:00'::TIME)
ORDER BY brec_fol_date,
brec_fol_b_focal_animid,
brec_time;
</pre>

=== Solution ===

Temporarily exclude the offending rows from the production conversion while retaining them in clean.brecord_notes. Correct the source times in Access after reviewing the original records. Do not clamp the values or weaken the production time constraints.

== (#117) BRECORD_NOTES text values do not satisfy production constraints ==

=== Problem ===

All production BRECORD_NOTES text columns are NOT NULL. Observation must also be trimmed of leading and trailing spaces. The remaining text columns permit the empty string to represent absent text but reject values consisting only of whitespace.

The Access source uses SQL NULL for absent text and contains some whitespace-only values. In the local conversion data examined on 2026-08-20, the rows otherwise eligible for conversion included 41 NULL observations and 14,349 untrimmed observations. The optional columns contained between 15,080 and 566,253 NULL values. Four comments, three Voc values, and three VocID values consisted only of whitespace.

=== Bad data ===

<pre>
SELECT brec_fol_date,
brec_fol_b_focal_animid,
brec_time,
field_name,
quote_nullable(field_value) AS field_value
FROM clean.brecord_notes
CROSS JOIN LATERAL (
VALUES ('Observation', brec_observation),
('Comments', brec_comments),
('Observer', brec_observer),
('Translator', brec_translator),
('TranscribedBy', brec_transcribed_by),
('Voc', brec_voc),
('VocID', brec_vocid),
('GroomingAggression', brec_grooming_aggressionflag),
('Duplicate', brec_duplicate_flag)
) AS source_text(field_name, field_value)
WHERE field_value IS NULL
OR (field_value <> '' AND btrim(field_value) = '')
OR (field_name = 'Observation'
AND field_value <> btrim(field_value))
ORDER BY brec_fol_date,
brec_fol_b_focal_animid,
brec_time,
field_name;
</pre>

=== Solution ===

Normalize only what the production constraints require at the final load boundary. Convert a NULL or whitespace-only value to the empty string. Trim Observation because its production column requires it, but preserve leading and trailing spaces in other nonempty source text. Keep the unchanged source-like values available in clean.brecord_notes.

== (#118) PANTGRUNT_EVENT rows have no recorded time ==

=== Problem ===

Some PANTGRUNT_EVENT rows have NULL in pg_time. EVENTS.Start and EVENTS.Stop are not nullable, but the absence of a recorded pantgrunt time is meaningful and must not cause the entire row to be discarded. In the local conversion data examined on 2026-08-21, 684 rows had no recorded time.

=== Bad data ===

<pre>
SELECT *
FROM clean.pantgrunt_event
WHERE pg_time IS NULL
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_b_actor_id,
pg_b_recipient_id;
</pre>

=== Solution ===

At the production load boundary, convert a NULL pg_time to the sdb_no_time value, midnight, and use that value for both EVENTS.Start and EVENTS.Stop. Extend the EVENTS constraints so pantgrunt events, like aggression events, may use sdb_no_time. Retain NULL in clean.pantgrunt_event so the clean schema continues to distinguish missing source data from an actual recorded time.

== (#119) PANTGRUNT_EVENT times may contain seconds ==

=== Problem ===

EVENTS times are stored to the minute and reject nonzero seconds. The Access pg_time value is initially represented as a timestamp, and merely converting it to a PostgreSQL time value does not remove seconds. The local clean data examined on 2026-08-21 contained no remaining nonzero seconds, but the pantgrunt workflow previously had no explicit normalization guaranteeing that result.

=== Bad data ===

<pre>
SELECT pg_date,
pg_fol_b_foc_id,
pg_time,
extract(second FROM pg_time) AS seconds
FROM clean.pantgrunt_event
WHERE extract(second FROM pg_time) <> 0
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time;
</pre>

=== Solution ===

During construction of the tidy schema, truncate pg_time to the minute before tidy.sql converts the column to TIME. Do not round the value. The pantgrunt sanity check rejects the conversion if a value containing seconds nevertheless reaches the clean schema.

== * (#120) PANTGRUNT_EVENT times fall outside the permitted event window ==

=== Problem ===

Except for sdb_no_time, production event times must be between 04:00 and 20:00, inclusive. In the local conversion data examined on 2026-08-21, 35 non-NULL pantgrunt times were outside this interval. Their correct values cannot be inferred during conversion.

=== Bad data ===

<pre>
SELECT *
FROM clean.pantgrunt_event
WHERE pg_time IS NOT NULL
AND (pg_time < '04:00'::TIME
OR pg_time > '20:00'::TIME)
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time,
pg_b_actor_id,
pg_b_recipient_id;
</pre>

=== Solution ===

Temporarily exclude these rows from the production conversion and report their count in the pantgrunt sanity check. Retain the rows unchanged in clean.pantgrunt_event. The project investigators must review and correct the source times in Access; the conversion must not clamp them or weaken the production time constraints.

== * (#121) PANTGRUNT_EVENT rows name the same actor and recipient ==

=== Problem ===

A production pantgrunt is a dyadic event involving two different participants. Some source rows identify the same animal as both pg_b_actor_id and pg_b_recipient_id. In the local conversion data examined on 2026-08-21, 11 rows had the same trimmed actor and recipient value.

=== Bad data ===

<pre>
SELECT *
FROM clean.pantgrunt_event
WHERE BTRIM(pg_b_actor_id) = BTRIM(pg_b_recipient_id)
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time,
pg_b_actor_id;
</pre>

=== Solution ===

Temporarily exclude these rows and report their count in the pantgrunt sanity check. The project investigators must determine the correct actor or recipient and correct the source records in Access.

== (#122) PANTGRUNT_EVENT rows have no data source ==

=== Problem ===

PANTGRUNTS.Source is required and must reference PG_SOURCES, but some source pantgrunt rows have a NULL, empty, or whitespace-only pg_data_source. In the local conversion data examined on 2026-08-21, 109 rows had no data source.

=== Bad data ===

<pre>
SELECT *
FROM clean.pantgrunt_event
WHERE NULLIF(BTRIM(pg_data_source), '') IS NULL
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time,
pg_b_actor_id,
pg_b_recipient_id;
</pre>

=== Solution ===

During construction of the clean schema, normalize source codes by trimming and converting them to uppercase. Convert a missing source to NODATA. Populate PG_SOURCES from the distinct normalized source values and give NODATA the description "No pantgrunt data source was recorded". Load NODATA into PANTGRUNTS.Source for affected rows.

== (#123) PANTGRUNT_EVENT rows have no extractor ==

=== Problem ===

PANTGRUNTS.EnteredBy is required and must reference an active PEOPLE row, but some source rows have a NULL, empty, or whitespace-only pg_extracted_by value. In the local conversion data examined on 2026-08-21, 2,223 rows had no recorded extractor.

=== Bad data ===

<pre>
SELECT *
FROM clean.pantgrunt_event
WHERE NULLIF(BTRIM(pg_extracted_by), '') IS NULL
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time,
pg_b_actor_id,
pg_b_recipient_id;
</pre>

=== Solution ===

During construction of the clean schema, change a NULL, empty, or whitespace-only pg_extracted_by value to UNK. The established UNK row is copied from clean.people to codes.people as an active person and is loaded into PANTGRUNTS.EnteredBy for affected eligible rows.

== (#124) PANTGRUNT_EVENT extractor values are missing from PEOPLE ==

=== Problem ===

Many nonempty pg_extracted_by values do not exactly match a clean.people person value. The local conversion data examined on 2026-08-21 had 26,334 rows without an exact PEOPLE match when missing values were included, and 24,111 nonmissing rows without a match after trimming. Some extractor names also occur with case variants, such as Anne and ANNE, while codes.people requires case-insensitive uniqueness.

=== Bad data ===

<pre>
SELECT BTRIM(pg.pg_extracted_by) AS extracted_by,
COUNT(*) AS row_count,
ARRAY_AGG(DISTINCT pg.pg_extracted_by ORDER BY pg.pg_extracted_by) AS source_spellings
FROM clean.pantgrunt_event AS pg
WHERE NULLIF(BTRIM(pg.pg_extracted_by), '') IS NOT NULL
AND NOT EXISTS (
SELECT 1
FROM clean.people AS p
WHERE p.person = BTRIM(pg.pg_extracted_by))
GROUP BY BTRIM(pg.pg_extracted_by)
ORDER BY row_count DESC,
extracted_by;
</pre>

=== Solution ===

During construction of the clean schema, trim nonempty pantgrunt extractor values and add one deterministic spelling of each case-insensitive value to clean.people when no case-insensitive PEOPLE match already exists. Prefer an existing PEOPLE spelling, and otherwise prefer a mixed-case source spelling over an all-uppercase or all-lowercase spelling. Rewrite pg_extracted_by to the exact spelling stored in clean.people. The ordinary support-table loader then copies these rows to codes.people with active set to true, which is required because new PANTGRUNTS rows may reference only active people.

== (#125) PANTGRUNT_EVENT two-sided flags use multiple encodings ==

=== Problem ===

The pg_two_sided_flag determines whether the two participants receive directed Actor and Actee roles or two Mutual roles. The Access data use NULL, N, Y, X, lowercase x, and potentially questionable yes/no forms rather than a single canonical encoding.

=== Bad data ===

<pre>
SELECT pg_two_sided_flag,
quote_nullable(pg_two_sided_flag) AS quoted_value,
COUNT(*) AS row_count
FROM clean.pantgrunt_event
GROUP BY pg_two_sided_flag
ORDER BY row_count DESC,
pg_two_sided_flag;
</pre>

=== Solution ===

Normalize NULL, the empty string, N, N?, and ? to N in the clean schema. Normalize Y, Y?, X, and lowercase x to Y. During loading, Y creates two Mutual roles; N creates an Actor role for pg_b_actor_id and an Actee role for pg_b_recipient_id. The sanity check stops conversion if an unrecognized value remains.

== (#126) PANTGRUNT_EVENT multiple-participant flags use noncanonical encodings ==

=== Problem ===

PANTGRUNTS.MultiActors and PANTGRUNTS.MultiRecipients are required Boolean values, while the corresponding Access fields are nullable text flags. The local conversion data include a lowercase x in pg_multiple_recipient_flag, and future dumps may include questionable yes/no forms.

=== Bad data ===

<pre>
SELECT flag_name,
flag_value,
COUNT(*) AS row_count
FROM clean.pantgrunt_event
CROSS JOIN LATERAL (
VALUES ('pg_multiple_actor_flag', pg_multiple_actor_flag),
('pg_multiple_recipient_flag', pg_multiple_recipient_flag)
) AS flags(flag_name, flag_value)
GROUP BY flag_name,
flag_value
ORDER BY flag_name,
row_count DESC,
flag_value;
</pre>

=== Solution ===

Normalize NULL, the empty string, N, N?, and ? to the empty string in the clean schema and load them as false. Normalize Y, Y?, X, and lowercase x to X and load them as true. The sanity check stops conversion if an unrecognized value remains.

== * (#127) PANTGRUNT_EVENT participants are missing from BIOGRAPHY_DATA ==

=== Problem ===

Every pantgrunt actor and recipient must be represented by a BIOGRAPHY_DATA row before a corresponding ROLES row can be created. In the local conversion data examined on 2026-08-21, 165 rows had an actor missing from BIOGRAPHY_DATA and 609 rows had a recipient missing from BIOGRAPHY_DATA. A row can be counted in both categories.

=== Bad data ===

<pre>
SELECT pg.*,
participant.source_column,
participant.animid
FROM clean.pantgrunt_event AS pg
CROSS JOIN LATERAL (
VALUES ('pg_b_actor_id', pg.pg_b_actor_id),
('pg_b_recipient_id', pg.pg_b_recipient_id)
) AS participant(source_column, animid)
WHERE NOT EXISTS (
SELECT 1
FROM clean.biography AS b
WHERE b.b_animid = BTRIM(participant.animid))
ORDER BY pg.pg_date,
pg.pg_fol_b_foc_id,
pg.pg_time,
participant.source_column,
participant.animid;
</pre>

=== Solution ===

Temporarily exclude any row whose trimmed actor or recipient does not have a matching BIOGRAPHY_DATA animal. Report actor and recipient counts separately in the sanity check. The project investigators must identify the intended individuals and correct the source records in Access.

== (#128) PANTGRUNT_EVENT focal/date keys have no corresponding follow ==

=== Problem ===

A pantgrunt event must relate to a WATCHES row. The normal B-record watches are created from clean.follow, but many pantgrunt focal/date keys have no matching follow. The local conversion data examined on 2026-08-21 contained 2,443 normalized focal/date keys without a B-record watch before other pantgrunt exclusions were applied.

=== Bad data ===

<pre>
SELECT COALESCE(NULLIF(BTRIM(pg.pg_fol_b_foc_id), ''), 'NONE') AS watch_animid,
pg.pg_date,
COUNT(*) AS pantgrunt_rows
FROM clean.pantgrunt_event AS pg
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow AS f
WHERE f.fol_b_animid = BTRIM(pg.pg_fol_b_foc_id)
AND f.fol_date = pg.pg_date)
GROUP BY COALESCE(NULLIF(BTRIM(pg.pg_fol_b_foc_id), ''), 'NONE'),
pg.pg_date
ORDER BY pg.pg_date,
watch_animid;
</pre>

=== Solution ===

Prefer and reuse an existing B-record WATCHES row having the same focal and date. If no such watch exists for an otherwise eligible pantgrunt, reuse an existing Other watch or create a WATCHES row with Type Other. Use the pantgrunt community for a newly created watch. This avoids falsely representing an ad-hoc observation as a follow and avoids the missing-arrival warnings associated with B-record watches. The source association can be corrected later without discarding the converted pantgrunt.

== (#129) PANTGRUNT_EVENT rows have no focal animal ID ==

=== Problem ===

Some pantgrunt records are ad-hoc observations and contain a NULL, empty, or whitespace-only pg_fol_b_foc_id. A WATCHES row requires an animal ID, but assigning the unknown individual UNK would incorrectly imply that a focal existed and merely could not be identified. In the local conversion data examined on 2026-08-21, 5,136 rows had no focal value.

=== Bad data ===

<pre>
SELECT *
FROM clean.pantgrunt_event
WHERE NULLIF(BTRIM(pg_fol_b_foc_id), '') IS NULL
ORDER BY pg_date,
pg_time,
pg_b_actor_id,
pg_b_recipient_id;
</pre>

=== Solution ===

For WATCHES resolution only, normalize an absent pantgrunt focal to the established no-focal AnimID NONE. Create or reuse an Other watch for NONE and the pantgrunt date. Leave the original blank focal value unchanged in clean.pantgrunt_event.

== * (#130) PANTGRUNT_EVENT focal/date keys contain conflicting communities ==

'''Status: Partially resolved; unresolved for twelve no-focal/date keys. The current loader exclusion is broader than necessary.'''

=== Problem ===

Every pantgrunt is related through EVENTS to one supporting WATCHES row. WATCHES records one CommID, and only one watch of a given Type may exist for a particular Date and AnimID. PANTGRUNTS also records its own CommID. This duplication is intentional: PANTGRUNTS.CommID preserves the community recorded with the pantgrunt in Access, while WATCHES.CommID records the community of the supporting follow or watch period. The two values are permitted to disagree, although the production database reports the disagreement as a warning.

The local conversion data examined on 2026-08-21 contain 15 normalized focal/date keys with both KK and MT pantgrunt communities. These keys cover 104 pantgrunt rows. They fall into two materially different categories:

* Three named-focal keys, covering 48 pantgrunt rows, already have a corresponding follow. The follow supplies an unambiguous B-record WATCHES row whose community is KK. These rows can be converted without choosing or discarding a pantgrunt community: reuse the existing watch and preserve each source value in PANTGRUNTS.CommID. Pantgrunts recorded as MT will appropriately produce community-mismatch warnings.
* Twelve keys, covering 56 pantgrunt rows, have no focal animal ID and no corresponding follow. The conversion normalizes the absent focal to NONE for watch resolution. Each date contains both KK and MT pantgrunts, but the conversion can create only one Other watch for the combination of NONE and that date. There is no source-supported way to decide whether that WATCHES row should contain KK or MT.

The community conflict therefore does not itself prevent a PANTGRUNTS row from storing its source community. The unresolved problem is selecting the canonical WATCHES.CommID when a supporting watch must be created and there is no follow or focal context from which to determine it.

=== Bad data ===

The following query shows every conflicting key, any community supplied by a corresponding follow, the pantgrunt communities, and the number of affected pantgrunt rows:

<pre>
WITH conflict_keys AS (
SELECT COALESCE(NULLIF(BTRIM(pg_fol_b_foc_id), ''), 'NONE') AS watch_animid,
pg_date
FROM clean.pantgrunt_event
GROUP BY COALESCE(NULLIF(BTRIM(pg_fol_b_foc_id), ''), 'NONE'),
pg_date
HAVING COUNT(DISTINCT BTRIM(pg_cl_community_id)) > 1
)
SELECT conflict_keys.watch_animid,
conflict_keys.pg_date,
follow.fol_cl_community_id AS follow_community,
ARRAY_AGG(DISTINCT BTRIM(pantgrunt_event.pg_cl_community_id)
ORDER BY BTRIM(pantgrunt_event.pg_cl_community_id)) AS pantgrunt_communities,
COUNT(*) AS pantgrunt_rows
FROM conflict_keys
JOIN clean.pantgrunt_event
ON COALESCE(NULLIF(BTRIM(pantgrunt_event.pg_fol_b_foc_id), ''), 'NONE') = conflict_keys.watch_animid
AND pantgrunt_event.pg_date = conflict_keys.pg_date
LEFT JOIN clean.follow
ON BTRIM(follow.fol_b_animid) = conflict_keys.watch_animid
AND follow.fol_date = conflict_keys.pg_date
GROUP BY conflict_keys.watch_animid,
conflict_keys.pg_date,
follow.fol_cl_community_id
ORDER BY conflict_keys.pg_date,
conflict_keys.watch_animid;
</pre>

=== Examples with named focals ===

{| class="wikitable"
! Focal
! Date
! Follow community
! Pantgrunt communities
! Pantgrunt rows
|-
| FD
| 1996-09-08
| KK
| KK and MT
| 11
|-
| ZS
| 2013-11-22
| KK
| KK and MT
| 6
|-
| DL
| 2013-11-29
| KK
| KK and MT
| 31
|}

For example, the FD records can all use the existing B-record watch for FD on 1996-09-08. That watch remains associated with KK. Each pantgrunt retains either KK or MT in PANTGRUNTS.CommID. The MT records disagree with the watch community, but this is a documented and intentionally preserved source inconsistency rather than a conversion failure.

=== Examples without a focal ===

{| class="wikitable"
! Watch AnimID used by conversion
! Date
! Pantgrunt communities
! Pantgrunt rows
|-
| NONE
| 1986-01-11
| KK and MT
| 8
|-
| NONE
| 1986-09-10
| KK and MT
| 16
|-
| NONE
| 1986-10-30
| KK and MT
| 3
|-
| NONE
| 1987-09-26
| KK and MT
| 7
|-
| NONE
| 1992-11-21
| KK and MT
| 3
|-
| NONE
| 1993-12-25
| KK and MT
| 2
|-
| NONE
| 1994-01-19
| KK and MT
| 3
|-
| NONE
| 1994-05-26
| KK and MT
| 2
|-
| NONE
| 1994-07-07
| KK and MT
| 2
|-
| NONE
| 1994-08-01
| KK and MT
| 3
|-
| NONE
| 1994-08-08
| KK and MT
| 4
|-
| NONE
| 1994-10-07
| KK and MT
| 3
|}

For example, the eight records on 1986-01-11 require a supporting Other watch with AnimID NONE. The WATCHES uniqueness rule permits only one such Other watch on that date. Assigning KK would be unsupported for the MT observations, while assigning MT would be unsupported for the KK observations. Creating two Other watches distinguished only by community is not permitted by the current WATCHES key.

=== Solution ===

Refine the conversion so that the three named-focal keys reuse their existing B-record watches and all 48 pantgrunt rows are loaded. Preserve the source community in PANTGRUNTS.CommID and accept the documented warning when it differs from the B-record watch community.

Continue to exclude only the 56 pantgrunt rows belonging to the twelve no-focal/date keys. Report those twelve keys in the sanity check. The investigators must determine whether each date represents one observation period with incorrect community coding, two distinct observation periods that the current WATCHES key cannot represent, or records with incorrect dates or focal values.

Until that refinement is implemented, the current loader conservatively excludes all 15 conflicting keys and therefore excludes 48 otherwise convertible named-focal rows in addition to the 56 genuinely ambiguous no-focal rows.

== * (#131) PANTGRUNT_EVENT observer values have no lossless production destination ==

'''Status: Unresolved; intentionally retained only in the clean schema.'''

=== Problem ===

The pg_observer field is populated only with JF in the local conversion data examined on 2026-08-21, affecting 1,696 rows across 221 normalized watch keys. PANTGRUNTS has no observer column. FOLLOW_OBSERVERS describes observers assigned to follow periods, so creating FOLLOW_OBSERVERS rows from pantgrunt records would require deciding whether pg_observer represents the same concept and period.

=== Bad data ===

<pre>
SELECT pg_observer,
COUNT(*) AS pantgrunt_rows,
COUNT(DISTINCT (COALESCE(NULLIF(BTRIM(pg_fol_b_foc_id), ''), 'NONE'), pg_date)) AS watch_keys
FROM clean.pantgrunt_event
WHERE pg_observer IS NOT NULL
GROUP BY pg_observer
ORDER BY pg_observer;
</pre>

=== Solution ===

Do not convert pg_observer into FOLLOW_OBSERVERS or another production table without an investigator-approved semantic mapping. Retain the values in clean.pantgrunt_event for later review. The pantgrunt loader explicitly documents that this field is omitted.

== * (#132) PANTGRUNT_EVENT participants occur outside their biography intervals ==

=== Problem ===

ROLES requires an event participant's date to fall between BIOGRAPHY_DATA.EntryDate and BIOGRAPHY_DATA.DepartDate, inclusive. Some pantgrunt actors or recipients have a biography row but occur before entry or after departure. In the local conversion data examined on 2026-08-21, 792 rows involved at least one participant outside this interval.

=== Bad data ===

<pre>
SELECT pg.*,
participant.source_column,
participant.animid,
b.b_entrydate,
b.b_departdate,
CASE
WHEN pg.pg_date < b.b_entrydate THEN 'BEFORE ENTRY'
WHEN pg.pg_date > b.b_departdate THEN 'AFTER DEPARTURE'
END AS date_problem
FROM clean.pantgrunt_event AS pg
CROSS JOIN LATERAL (
VALUES ('pg_b_actor_id', pg.pg_b_actor_id),
('pg_b_recipient_id', pg.pg_b_recipient_id)
) AS participant(source_column, animid)
JOIN clean.biography AS b
ON b.b_animid = BTRIM(participant.animid)
WHERE pg.pg_date NOT BETWEEN b.b_entrydate AND b.b_departdate
ORDER BY pg.pg_date,
pg.pg_fol_b_foc_id,
pg.pg_time,
participant.source_column,
participant.animid;
</pre>

=== Solution ===

Temporarily exclude affected rows and report their count in the pantgrunt sanity check. The project investigators must determine whether the pantgrunt date, participant identity, or biography interval should be corrected in Access. Do not weaken the ROLES temporal constraints.

== * (#133) PANTGRUNT_EVENT contains invalid nonempty focal animal IDs ==

=== Problem ===

An Other WATCHES row created for a pantgrunt requires a valid BIOGRAPHY_DATA animal ID. Some pg_fol_b_foc_id values are nonempty but do not match a biography animal and therefore cannot be treated as the no-focal value NONE. In the local conversion data examined on 2026-08-21, 70 rows contained such focal values.

=== Bad data ===

<pre>
SELECT pg.*
FROM clean.pantgrunt_event AS pg
WHERE NULLIF(BTRIM(pg.pg_fol_b_foc_id), '') IS NOT NULL
AND NOT EXISTS (
SELECT 1
FROM clean.biography AS b
WHERE b.b_animid = BTRIM(pg.pg_fol_b_foc_id))
ORDER BY pg.pg_date,
pg.pg_fol_b_foc_id,
pg.pg_time,
pg.pg_b_actor_id,
pg.pg_b_recipient_id;
</pre>

=== Solution ===

Temporarily exclude these rows and report their count in the sanity check. The project investigators must identify whether each value is a malformed focal ID, descriptive text, or evidence that no focal existed, then correct Access accordingly. Do not automatically convert a nonempty value to NONE or UNK.

== (#134) PANTGRUNT_EVENT notes may be NULL ==

=== Problem ===

EVENTS.Notes is required but permits the empty string. The Access pantgrunt table uses NULL for absent notes. In the local conversion data examined on 2026-08-21, 7,339 rows had NULL pg_notes and no rows contained whitespace-only notes.

=== Bad data ===

<pre>
SELECT *
FROM clean.pantgrunt_event
WHERE pg_notes IS NULL
OR (pg_notes <> '' AND BTRIM(pg_notes) = '')
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time,
pg_b_actor_id,
pg_b_recipient_id;
</pre>

=== Solution ===

At the production load boundary, convert a NULL or whitespace-only pg_notes value to the empty string and otherwise preserve the source text unchanged. Retain the original value in clean.pantgrunt_event.

== * (#135) PANTGRUNT_EVENT year values have no lossless production destination ==

=== Problem ===

The production pantgrunt model obtains the event date through WATCHES and has no separate year column. Although pg_year appears redundant with pg_date, the local conversion data examined on 2026-08-21 contain 178 rows where pg_year differs from the year extracted from pg_date. Discarding the field as derived data would therefore lose a recorded discrepancy.

=== Bad data ===

''records''

<pre>
SELECT *
FROM clean.pantgrunt_event
WHERE pg_year IS NULL
OR pg_year IS DISTINCT FROM extract(year FROM pg_date)::INTEGER
ORDER BY pg_date,
pg_fol_b_foc_id,
pg_time,
pg_b_actor_id,
pg_b_recipient_id;
</pre>

''summary''

<pre>
SELECT pg_year,
extract(year FROM pg_date)::INTEGER AS date_year,
pg_year - extract(year FROM pg_date)::INTEGER AS year_difference,
count(*) AS row_count
FROM clean.pantgrunt_event
WHERE pg_year IS DISTINCT FROM
extract(year FROM pg_date)::INTEGER
GROUP BY pg_year,
extract(year FROM pg_date)::INTEGER
ORDER BY pg_year,
date_year;
</pre>

=== Solution ===

Do not load pg_year into another production column or silently replace pg_date. Retain pg_year in clean.pantgrunt_event for investigator review. The pantgrunt loader explicitly documents that the field is omitted pending a project decision about the 178 discrepancies.

== (#136) Some GROOM_SCANS direction codes have trailing spaces ==

=== Problem ===

There are 3 GROOM_SCANS records where GS_direction is 'G ' instead of 'G'. The trailing Access padding prevents an exact match with the valid one-character direction code.

=== Bad Data ===

Cleaned in the clean schema, so query the easy schema.

<pre>
SELECT '"' || gs_direction || '"' AS untrimmed_direction,
'"' || BTRIM(gs_direction) || '"' AS trimmed_direction,
count(*) AS row_count
FROM easy.groom_scans
WHERE gs_direction IS DISTINCT FROM BTRIM(gs_direction)
GROUP BY gs_direction,
BTRIM(gs_direction)
ORDER BY gs_direction;
</pre>

This reports 3 rows having "G ", which normalizes to "G".

=== Solution ===

Trim leading and trailing whitespace from GS_direction in the clean schema. This preserves all source rows and allows the direction codes to be mapped normally by the production loader.

Resolved by commit 3ad4f5e1e2355895e5e465c4b8049197c8dce295

== * (#137) GROOM_BOUT focal/date keys have no corresponding follow ==

=== Problem ===

A grooming event must relate to a WATCHES row. Normal B-record watches are created from clean.follow, but 1,536 current GROOM_BOUT rows have no follow with the same focal and date after animal-ID whitespace normalization. GROOM_BOUT has no community column, so the loader cannot create a watch directly from the source row.

Of these rows, 1,399 have exactly one COMMUNITY_MEMBERSHIP row covering the observation date. The remaining 137 do not have exactly one dated membership and must not be assigned a community by inference.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow
WHERE follow.fol_b_animid = gb.grm_fol_b_focal_animid
AND follow.fol_date = gb.grm_fol_date)
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid;
</pre>

The temporarily excluded subset is identified by:

<pre>
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow
WHERE follow.fol_b_animid = gb.grm_fol_b_focal_animid
AND follow.fol_date = gb.grm_fol_date)
AND 1 <> (
SELECT count(*)
FROM clean.community_membership
WHERE community_membership.cm_b_animid =
gb.grm_fol_b_focal_animid
AND gb.grm_fol_date BETWEEN
community_membership.cm_start_date
AND community_membership.cm_end_date)
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid;
</pre>

=== Solution ===

As established for ad-hoc observations in Problem #128, prefer an existing B-record WATCHES row and otherwise reuse an existing Other watch for the focal and date. When no watch exists and exactly one COMMUNITY_MEMBERSHIP interval covers the observation date, create an Other watch using that membership's community. This preserves the grooming bout without falsely representing it as part of a follow.

Do not infer a community from unrelated observations when there is not exactly one dated membership. Temporarily exclude those 137 rows from the production groom-bout loader. Three overlap Problem #138, so this exclusion adds 134 rows to the excluded union. Investigators must determine the focal's community on the observation date or correct the missing follow or membership data in Access.

== * (#138) GROOM_BOUT initiator or terminator is not a member of the grooming pair ==

=== Problem ===

After applying the approved Problem #112 interpretation of U as unknown and trimming animal IDs under Problem #139, 2,149 GROOM_BOUT rows have a nonempty initiator or terminator animal ID that is neither the focal nor the recorded grooming partner. There are 1,776 initiator mismatches and 2,133 terminator mismatches, with overlap between those sets.

Production GROOMINGS.Initiator and GROOMINGS.Terminator values must reference ROLES.PID rows belonging to participants in the same grooming event. The source does not establish whether the mismatched value, the recorded partner, or another field is incorrect, so the loader cannot safely choose a role or add another participant.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
gb.grm_direction,
invalid_reference.source_column,
invalid_reference.animid AS offending_animid,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
CROSS JOIN LATERAL (
VALUES ('GRM_B_initiator_AnimId', gb.grm_b_initiator_animid),
('GRM_B_terminator_AnimId', gb.grm_b_terminator_animid)
) AS invalid_reference(source_column, animid)
WHERE invalid_reference.animid IS NOT NULL
AND invalid_reference.animid NOT IN (
gb.grm_fol_b_focal_animid,
gb.grm_b_partner_animid)
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
invalid_reference.source_column;
</pre>

=== Solution ===

Temporarily exclude affected rows from the production groom-bout loader. The project investigators must determine the correct grooming partner and the correct initiator or terminator in Access. Keep the production requirement that an initiator or terminator reference a participant role in the same grooming event.

== (#139) GROOM_BOUT animal IDs contain edge whitespace ==

=== Problem ===

There are 50 animal-ID values with leading or trailing whitespace across 49 GROOM_BOUT rows: 36 focal values, 9 partner values, 2 initiator values, and 3 terminator values. The first runtime failure used partner value <code>BE </code>, which does not reference BIOGRAPHY_DATA even though the trimmed value <code>BE</code> does.

All 36 focal and all 9 partner values identify existing BIOGRAPHY rows after trimming. Trimming is the established lossless conversion treatment for animal-ID edge whitespace.

=== Bad data ===

''note: querying easy because the values are normalized in clean''

<pre>
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
animal_id.source_column,
animal_id.animid,
BTRIM(animal_id.animid) AS trimmed_animid
FROM easy.groom_bout AS gb
CROSS JOIN LATERAL (
VALUES ('GRM_FOL_B_focal_AnimId', gb.grm_fol_b_focal_animid),
('GRM_B_partner_AnimId', gb.grm_b_partner_animid),
('GRM_B_initiator_AnimId', gb.grm_b_initiator_animid),
('GRM_B_terminator_AnimId', gb.grm_b_terminator_animid)
) AS animal_id(source_column, animid)
WHERE animal_id.animid IS DISTINCT FROM BTRIM(animal_id.animid)
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
animal_id.source_column;
</pre>

=== Solution ===

Trim leading and trailing whitespace from the four GROOM_BOUT animal-ID columns in the clean schema. Apply the Problem #112 conversion of initiator or terminator U values to SQL NULL after trimming. Do not change GRM_direction.

== * (#140) GROOM_BOUT focal or partner animal IDs are absent from BIOGRAPHY ==

=== Problem ===

After the lossless Problem #139 whitespace normalization, 412 GROOM_BOUT rows have a focal or grooming partner that is absent from BIOGRAPHY: 131 rows have an absent focal and 337 have an absent partner. Some rows occur in both counts. Production WATCHES.AnimID and ROLES.Participant values must reference BIOGRAPHY_DATA.AnimID.

The first otherwise eligible runtime failure is the 1992-10-13 PF/MGE bout. The source does not identify a production animal for MGE, so no sentinel or other animal can be substituted safely.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
missing_participant.source_column,
missing_participant.animid,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
CROSS JOIN LATERAL (
VALUES ('GRM_FOL_B_focal_AnimId', gb.grm_fol_b_focal_animid),
('GRM_B_partner_AnimId', gb.grm_b_partner_animid)
) AS missing_participant(source_column, animid)
WHERE NOT EXISTS (
SELECT 1
FROM clean.biography
WHERE biography.b_animid = missing_participant.animid)
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
missing_participant.source_column;
</pre>

=== Solution ===

Temporarily exclude affected rows from the production groom-bout loader. Of the 412 matches, 206 overlap Problems #137 or #138 and 206 are newly excluded. Investigators must identify the intended animals and correct the Access data. Keep the production foreign keys to BIOGRAPHY_DATA intact.

== * (#141) GROOM_BOUT participants are observed before entry into the study ==

=== Problem ===

Seven GROOM_BOUT rows involve a focal or partner before that animal's BIOGRAPHY.EntryDate: one focal occurrence and six partner occurrences. Production rejects a role dated before its participant entered the study. The first otherwise eligible failure observes SN on 1996-03-11, before its 1996-06-03 entry date.

The identifying predicate uses a strict less-than comparison, so an observation on EntryDate remains valid.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
early_participant.source_column,
early_participant.animid,
biography.b_entrydate,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
CROSS JOIN LATERAL (
VALUES ('GRM_FOL_B_focal_AnimId', gb.grm_fol_b_focal_animid),
('GRM_B_partner_AnimId', gb.grm_b_partner_animid)
) AS early_participant(source_column, animid)
JOIN clean.biography
ON biography.b_animid = early_participant.animid
WHERE gb.grm_fol_date < biography.b_entrydate
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
early_participant.source_column;
</pre>

=== Solution ===

Temporarily exclude affected rows from the production groom-bout loader. One overlaps Problems #137, #138, or #140, so this exclusion adds six rows to the excluded union. Investigators must correct the observation date, participant, or biography entry date in Access. Keep the production rule and its inclusive EntryDate boundary intact.

== * (#142) GROOM_BOUT participants are observed after departure from the study ==

=== Problem ===

Sixty GROOM_BOUT rows involve a focal or partner after that animal's BIOGRAPHY.DepartDate: five rows have an affected focal and 56 have an affected partner. Some rows occur in both counts. Production rejects a role dated after its participant departed the study. The first otherwise eligible failure observes FI on 1997-07-30, one day after its 1997-07-29 departure date.

The identifying predicate uses a strict greater-than comparison, so an observation on DepartDate remains valid.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
late_participant.source_column,
late_participant.animid,
biography.b_departdate,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
CROSS JOIN LATERAL (
VALUES ('GRM_FOL_B_focal_AnimId', gb.grm_fol_b_focal_animid),
('GRM_B_partner_AnimId', gb.grm_b_partner_animid)
) AS late_participant(source_column, animid)
JOIN clean.biography
ON biography.b_animid = late_participant.animid
WHERE gb.grm_fol_date > biography.b_departdate
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
late_participant.source_column;
</pre>

=== Solution ===

Temporarily exclude affected rows from the production groom-bout loader. Eleven overlap Problems #137, #138, #140, or #141, so this exclusion adds 49 rows to the excluded union. Investigators must correct the observation date, participant, or biography departure date in Access. Keep the production rule and its inclusive DepartDate boundary intact.

== * (#143) GROOM_BOUT other-partner flags have non-boolean values ==

=== Problem ===

GROOMINGS.Others is a required boolean, while GROOM_BOUT.GRM_other_partners contains several encodings. The established Access boolean-style convention treats NULL or the empty string as false, and the source also contains lowercase y and n values that differ only by case. There are 4,266 blank or NULL values and 561 lowercase values.

After those lossless boundary normalizations, nine rows retain values that cannot be mapped to a boolean: B in three rows, G in one row, M in two rows, and U in three rows. The first otherwise eligible runtime failure has U on the 2002-07-28 SA/SR bout.

=== Bad data ===

''summary before clean-schema normalization''

<pre>
SELECT COALESCE(grm_other_partners, '<NULL>') AS raw_value,
count(*) AS row_count
FROM easy.groom_bout
GROUP BY grm_other_partners
ORDER BY raw_value;
</pre>

''unmappable values in clean, the schema read by the production loader''

<pre>
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
gb.grm_other_partners,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
WHERE gb.grm_other_partners NOT IN ('Y', 'N')
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid;
</pre>

=== Solution ===

During construction of the clean schema, convert NULL and empty GRM_other_partners values to N and normalize lowercase y and n to uppercase. Temporarily exclude the nine rows whose normalized value is not Y or N. One overlaps Problems #137, #138, #140, #141, or #142, so this exclusion adds eight rows to the excluded union.

Investigators must determine whether B, G, M, and U mean that other grooming partners were present and correct the Access data. Keep GROOMINGS.Others required and boolean.

== * (#144) GROOM_BOUT times are outside the production observation window ==

=== Problem ===

Six GROOM_BOUT rows have start and stop times outside the production EVENTS range of 04:00 through 20:00 inclusive. All six violate both the start and stop constraints. The first runtime failure is the 2007-03-28 NUR/KS bout from 02:55 through 02:57.

The source does not establish whether these are valid nighttime observations or mistyped times. Changing the production observation window requires a separate decision.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_time_end,
gb.grm_b_partner_animid,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
WHERE gb.grm_time_begin < '04:00'::TIME
OR gb.grm_time_begin > '20:00'::TIME
OR (gb.grm_time_end IS NOT NULL
AND (gb.grm_time_end < '04:00'::TIME
OR gb.grm_time_end > '20:00'::TIME))
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid;
</pre>

=== Solution ===

Temporarily exclude the six rows from the production groom-bout loader. None overlap Problems #137, #138, #140, #141, #142, or #143. Investigators must confirm the intended times or decide separately whether the production observation window should change. Keep the current production constraints intact.

== * (#145) GROOM_BOUT records the same animal as focal and partner ==

=== Problem ===

Four GROOM_BOUT rows identify the same animal as both the focal and grooming partner. Production represents grooming as a dyadic event and requires each ROLES participant to be unique within an event, so the second role violates the unique participant-and-event constraint. The first failing row is the 2011-07-31 TOM/TOM bout, whose source problems text includes "WRONG FOCAL".

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid,
gb.grm_direction,
gb.grm_extracted_by,
gb.grm_problems,
gb.grm_comments
FROM clean.groom_bout AS gb
WHERE gb.grm_fol_b_focal_animid = gb.grm_b_partner_animid
ORDER BY gb.grm_fol_date,
gb.grm_fol_b_focal_animid,
gb.grm_time_begin,
gb.grm_b_partner_animid;
</pre>

=== Solution ===

Temporarily exclude the four rows from the production groom-bout loader. None overlap Problems #137, #138, #140, #141, #142, #143, or #144. Investigators must identify the intended focal or grooming partner and correct the Access data. Keep the production requirement that a participant occur at most once in an event.

== * (#146) GROOM_SCANS participants are observed after departure from the study ==

=== Problem ===

Sixty-four GROOM_SCANS rows involve a grooming participant after that animal's BIOGRAPHY.DepartDate: eight rows have an affected <code>GS_B_chimp1_AnimId</code> and 56 have an affected <code>GS_B_chimp2_AnimId</code>. Production rejects a role dated after its participant departed the study. The first runtime failure observes MT on 1978-05-23, after its 1974-11-01 departure date.

The identifying predicate uses a strict greater-than comparison, so an observation on DepartDate remains valid. Seventeen participant occurrences are on their departure date.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
late_participant.source_column,
late_participant.animid,
biography.b_departdate,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
CROSS JOIN LATERAL (
VALUES ('GS_B_chimp1_AnimId', gs.gs_b_chimp1_animid),
('GS_B_chimp2_AnimId', gs.gs_b_chimp2_animid)
) AS late_participant(source_column, animid)
JOIN clean.biography
ON biography.b_animid = late_participant.animid
WHERE gs.gs_date > biography.b_departdate
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
late_participant.source_column;
</pre>

=== Solution ===

Temporarily exclude the 64 affected rows from the production B-record groom-scan loader. Investigators must correct the observation date, participant, or biography departure date in Access. Keep the production rule and its inclusive DepartDate boundary intact.

== * (#147) GROOM_SCANS focal/date keys have no corresponding follow ==

=== Problem ===

A B-record groom scan must relate to a B-type WATCHES row. There are 673 current GROOM_SCANS rows whose focal and date have no matching clean.follow row. Of these, 665 rows have exactly one COMMUNITY_MEMBERSHIP interval covering the observation date and eight rows, comprising three focal/date keys, have no dated membership.

The first runtime failure is the 1978-06-29 ST scan at 14:15. Its source table identifies it as B-record data, and ST has exactly one community membership on that date, so the community is unambiguous even though the corresponding follow is absent.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow
WHERE follow.fol_b_animid = gs.gs_fol_b_focal_animid
AND follow.fol_date = gs.gs_date)
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
</pre>

The temporarily excluded subset is identified by:

<pre>
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE NOT EXISTS (
SELECT 1
FROM clean.follow
WHERE follow.fol_b_animid = gs.gs_fol_b_focal_animid
AND follow.fol_date = gs.gs_date)
AND 1 <> (
SELECT count(*)
FROM clean.community_membership
WHERE community_membership.cm_b_animid =
gs.gs_fol_b_focal_animid
AND gs.gs_date BETWEEN
community_membership.cm_start_date
AND community_membership.cm_end_date)
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
</pre>

=== Solution ===

Reuse an existing B-type WATCHES row when one was created by an earlier loader. Otherwise, when exactly one COMMUNITY_MEMBERSHIP interval covers the observation date, create a B watch using that membership's community. The GROOM_SCANS source identifies these observations as B-record data, so this does not invent the watch type.

Do not infer a community when no dated membership exists. Temporarily exclude the eight unresolved rows; they do not overlap Problem #146. Investigators must determine the focal's community on the observation date or correct the missing follow or membership data in Access.

== * (#148) GROOM_SCANS participants are observed before entry into the study ==

=== Problem ===

Four GROOM_SCANS rows involve a grooming participant before that animal's BIOGRAPHY.EntryDate. All four affected values are in <code>GS_B_chimp2_AnimId</code>. Production rejects a role dated before its participant entered the study. The first runtime failure observes KP on 1981-09-15, before its 1997-03-16 entry date.

The identifying predicate uses a strict less-than comparison, so an observation on EntryDate remains valid. No current GROOM_SCANS participant is observed exactly on its entry date.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
early_participant.source_column,
early_participant.animid,
biography.b_entrydate,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
CROSS JOIN LATERAL (
VALUES ('GS_B_chimp1_AnimId', gs.gs_b_chimp1_animid),
('GS_B_chimp2_AnimId', gs.gs_b_chimp2_animid)
) AS early_participant(source_column, animid)
JOIN clean.biography
ON biography.b_animid = early_participant.animid
WHERE gs.gs_date < biography.b_entrydate
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
early_participant.source_column;
</pre>

=== Solution ===

Temporarily exclude the four affected rows from the production B-record groom-scan loader. They do not overlap Problems #146 or #147. Investigators must correct the observation date, participant, or biography entry date in Access. Keep the production rule and its inclusive EntryDate boundary intact.

== * (#149) GROOM_SCANS times are outside the production observation window ==

=== Problem ===

Sixty-one GROOM_SCANS rows have times outside the production EVENTS range of 04:00 through 20:00 inclusive. The values range from 00:00 through 23:50. The first runtime failure is the 1982-01-13 PI scan at 03:30.

The source does not establish whether these are valid nighttime observations or mistyped times. Changing the production observation window requires a separate decision. No current GROOM_SCANS row occurs exactly at 04:00 or 20:00.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE gs.gs_time < '04:00'::TIME
OR gs.gs_time > '20:00'::TIME
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
</pre>

=== Solution ===

Temporarily exclude the 61 affected rows from the production B-record groom-scan loader. They do not overlap Problems #146, #147, or #148. Investigators must confirm the intended times or decide separately whether the production observation window should change. Keep the current production constraints intact.

== * (#150) GROOM_SCANS unknown direction cannot use the production UNKPair role ==

=== Problem ===

Thirty-one GROOM_SCANS rows have <code>GS_direction = 'U'</code>. The loader maps this source code to paired <code>UNKPair</code> roles, which represent a dyadic interaction whose direction is unknown. The current production <code>roles_func</code> and EVENTS documentation permit Actor, Actee, and Mutual roles for GSCAN events but reject <code>UNKPair</code>.

The first runtime failure is the 1982-02-07 ST scan at 11:55 with participants FF and GB. Its source comment says "NO DIRECTION INDICATED ON TIKI OR ON B RECORDS". None of the 31 rows overlaps Problems #146 through #149.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE gs.gs_direction = 'U'
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
</pre>

=== Solution ===

Temporarily exclude the 31 source rows while retaining the current production role restriction. The investigator does not want structural schema changes in the B-record loader commit. Flag permitting paired <code>UNKPair</code> roles for GSCAN events, with the corresponding EVENTS documentation change, as a structural problem for the next iteration.

Do not convert <code>U</code> to Mutual because unknown direction does not establish symmetric grooming.

== * (#151) GROOM_SCANS participants are absent from BIOGRAPHY ==

=== Problem ===

Two hundred twenty-five GROOM_SCANS rows have a grooming participant absent from BIOGRAPHY: 33 affected values are in <code>GS_B_chimp1_AnimId</code> and 192 are in <code>GS_B_chimp2_AnimId</code>. Production ROLES.Participant must reference BIOGRAPHY_DATA.AnimID. The first runtime failure is the 1982-03-09 FF scan at 15:45, whose second participant is TP.

The source does not identify a production animal that can safely replace an absent participant.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
missing_participant.source_column,
missing_participant.animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
CROSS JOIN LATERAL (
VALUES ('GS_B_chimp1_AnimId', gs.gs_b_chimp1_animid),
('GS_B_chimp2_AnimId', gs.gs_b_chimp2_animid)
) AS missing_participant(source_column, animid)
WHERE NOT EXISTS (
SELECT 1
FROM clean.biography
WHERE biography.b_animid = missing_participant.animid)
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
missing_participant.source_column;
</pre>

=== Solution ===

Temporarily exclude the 225 affected rows from the production B-record groom-scan loader. They do not overlap Problems #146 through #150. Investigators must identify the intended animals and correct the Access data. Keep the production foreign key to BIOGRAPHY_DATA intact.

== * (#152) GROOM_SCANS direction values cannot be mapped to production roles ==

=== Problem ===

After the Problem #136 whitespace normalization, 48 GROOM_SCANS rows have a direction other than G, R, M, or U: 43 are NULL, one is F, one is H, and three are T. The first runtime failure that reached the direction mapping has H on the 1989-06-22 GB scan at 08:15.

The source does not establish which participant groomed the other, whether grooming was mutual, or whether direction was unknown. The loader cannot choose production roles safely.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE COALESCE(gs.gs_direction, '') NOT IN ('G', 'R', 'M', 'U')
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
</pre>

=== Solution ===

Temporarily exclude the 48 affected rows from the production B-record groom-scan loader. Three overlap Problems #146 through #151, so this exclusion adds 45 rows to the excluded union. Investigators must determine the intended direction and correct the Access data. Do not infer roles from participant order or map missing direction to Mutual or unknown without a documented decision.

== * (#153) GROOM_SCANS records the same animal as both participants ==

=== Problem ===

Twelve GROOM_SCANS rows identify the same animal in <code>GS_B_chimp1_AnimId</code> and <code>GS_B_chimp2_AnimId</code>. Production represents groom scans as dyadic events and requires each ROLES participant to be unique within an event. The first runtime failure is the 2015-03-16 NUR/NUR scan at 13:00, whose source comment says "Same ID for chimp1 and chimp2".

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE gs.gs_b_chimp1_animid = gs.gs_b_chimp2_animid
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
</pre>

=== Solution ===

Temporarily exclude the 12 affected rows from the production B-record groom-scan loader. They do not overlap Problems #146 through #152. Investigators must identify the intended first or second participant, or decide separately how self-grooming should be represented. Keep the production requirement that a participant occur at most once in an event.

== * (#154) GROOM_SCANS records have no observation time ==

=== Problem ===

Eight GROOM_SCANS rows have NULL <code>GS_time</code> values. Production EVENTS requires a time, and all eight source comments say "no time". None of the rows overlaps Problems #146 through #153.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_time,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid,
gs.gs_direction,
gs.gs_extracted_by,
gs.gs_comments
FROM clean.groom_scans AS gs
WHERE gs.gs_time IS NULL
ORDER BY gs.gs_date,
gs.gs_fol_b_focal_animid,
gs.gs_b_chimp1_animid,
gs.gs_b_chimp2_animid;
</pre>

=== Solution ===

Temporarily exclude the eight affected rows from the production B-record groom-scan loader. Investigators must recover the observation times from another source or document a separate policy for missing event times. Do not invent a time from the scan date, record order, or neighboring observations.

== (#155) FERTILITY boundary-type codes need production support rows ==

=== Problem ===

The 168 <code>clean.fertility</code> rows use four <code>StartType</code> values and four <code>StopType</code>
values. Production FERTILITY requires these values to reference
FERTILITY_STARTS and FERTILITY_STOPS, but both support tables are empty.

The fertility domains exactly match the BIOGRAPHY_DATA boundary-type domains:
<code>B</code>, <code>C</code>, <code>I</code>, and <code>O</code> match ENTRYTYPES, while <code>D</code>, <code>E</code>, <code>O</code>, and <code>P</code> match
DEPARTTYPES.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT 'StartType' AS source_column,
fertility.starttype AS code,
COUNT(*) AS rows
FROM clean.fertility AS fertility
GROUP BY fertility.starttype
UNION ALL
SELECT 'StopType',
fertility.stoptype,
COUNT(*)
FROM clean.fertility AS fertility
GROUP BY fertility.stoptype
ORDER BY source_column, code;
</pre>

The refreshed 2026-09-13 source contains StartType <code>B</code> (70 rows), <code>C</code> (56),
<code>I</code> (34), and <code>O</code> (8), and StopType <code>D</code> (74), <code>E</code> (1), <code>O</code> (75), and <code>P</code>
(18).

=== Solution ===

The project approved using the corresponding ENTRYTYPES and DEPARTTYPES
meanings for fertility boundary types. Populate FERTILITY_STARTS with <code>B</code>
(Birth), <code>C</code> (Start of confirmed AnimID), <code>I</code> (Immigration), and <code>O</code>
(Initiation of close observation). Populate FERTILITY_STOPS with <code>D</code> (Death),
<code>E</code> (Emigration), <code>O</code> (End of observation; Present in the most recent census),
and <code>P</code> (Permanent disappearance).

This is a support-table decision and excludes no source rows.

== (#156) FERTILITY source intervals require ordered dates ==

=== Problem ===

Production FERTILITY does not have a table constraint requiring <code>StartDate <=
StopDate</code>. Loading a reversed source interval would therefore preserve an
invalid period without necessarily producing a database error.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT fertility.studyid,
fertility.animalid,
fertility.startdate,
fertility.starttype,
fertility.stopdate,
fertility.stoptype
FROM clean.fertility AS fertility
WHERE fertility.startdate > fertility.stopdate
ORDER BY fertility.studyid,
fertility.animalid,
fertility.startdate,
fertility.stopdate;
</pre>

The refreshed 2026-09-13 source contains zero reversed intervals.

=== Solution ===

Enforce <code>StartDate <= StopDate</code> in the read-only fertility conversion sanity
gate. Do not add a production table constraint as part of this conversion.
Abort before writing any production rows if a future source snapshot contains
a reversed interval.

This decision excludes no source rows.

== * (#157) Selected FERTILITY periods cross BIOGRAPHY_DATA boundaries ==

=== Problem ===

Nine <code>clean.fertility</code> rows extend beyond one or more BIOGRAPHY_DATA date
boundaries. Production does not require fertility periods to remain within
BirthDate, EntryDate, and DepartDate, so the database does not establish
whether these differences are valid history or source errors.

Investigators reviewed the nine rows and selected six for temporary exclusion:
AR, BH, GLIB1, MAM, NV, and RAF. The remaining boundary differences for SAF,
SHO, and SN1 are approved for conversion and are not excluded.

=== Bad data ===

''note: querying clean, the schema read by the production loader; all six source fields identify each approved exclusion''

<pre>
SELECT fertility.studyid,
fertility.animalid,
fertility.startdate,
fertility.starttype,
fertility.stopdate,
fertility.stoptype
FROM clean.fertility AS fertility
WHERE (fertility.studyid,
fertility.animalid,
fertility.startdate,
fertility.starttype,
fertility.stopdate,
fertility.stoptype)
IN ((4, 'AR', DATE '1985-07-11', 'B', DATE '1987-07-19', 'D'),
(4, 'BH', DATE '1971-07-05', 'B', DATE '1971-09-20', 'D'),
(4, 'GLIB1', DATE '2011-07-11', 'B', DATE '2012-01-21', 'D'),
(4, 'MAM', DATE '2004-02-12', 'B', DATE '2011-12-07', 'D'),
(4, 'NV', DATE '1965-09-15', 'I', DATE '1975-03-12', 'D'),
(4, 'RAF', DATE '1986-10-21', 'C', DATE '1996-03-27', 'D'))
ORDER BY fertility.animalid,
fertility.startdate;
</pre>

The query returns exactly six rows. Because this is a new FERTILITY loader,
none overlaps an earlier source-row exclusion. The distinct exclusion union is
six rows, leaving 162 eligible rows from the refreshed 168-row source.

=== Solution ===

Temporarily exclude the six identified rows from the production FERTILITY
loader. Keep the full six-column predicate adjacent to its diagnostic query so
the exclusion cannot broaden silently. Investigators must correct the Access
fertility or biography dates, or approve a later interpretation that permits
these periods, before the rows are restored.

Do not exclude SAF, SHO, or SN1. Their boundary differences were reviewed and
approved for conversion.

== (#158) FERTILITY StudyId 4 has no production STUDIES row ==

=== Problem ===

All 168 <code>clean.fertility</code> rows have integer <code>StudyId</code> 4. Production
FERTILITY.Study is text and must reference STUDIES, but the follow-derived
support data contains no Study code <code>4</code>. The Access snapshot contains no study
lookup table, description, foreign key, or imported metadata that gives StudyId
4 a textual name.

=== Bad data ===

''note: querying clean, the schema read by the production loader''

<pre>
SELECT fertility.studyid,
COUNT(*) AS rows
FROM clean.fertility AS fertility
GROUP BY fertility.studyid
ORDER BY fertility.studyid;
</pre>

The refreshed 2026-09-13 source returns only StudyId 4, with 168 rows.

=== Solution ===

Preserve the source value without inventing unsupported semantics. Add Study
code <code>4</code> to STUDIES with description <code>4</code>, and map <code>clean.fertility.studyid</code> to
<code>FERTILITY.Study</code> using <code>studyid::text</code> at the load boundary.

This support and representation decision excludes no source rows.

== (#159) FOOD_LOOKUP status flags need explicit conversion semantics ==

=== Problem ===

<code>clean.food_lookup</code> contains <code>FL_exclude</code> and
<code>FL_unverified</code> flags, but the mung support and event loaders ignored
both. Loading all lookup rows would admit names explicitly marked for
exclusion; treating unverified names the same way would discard data without
investigator approval.

=== Bad data ===

''The following is a dated diagnostic. Rerun it against the refreshed clean
schema and record exact referring rows before implementing exclusions.''

<pre>
WITH normalized_lookup AS (
SELECT UPPER(BTRIM(fl_local_food_name)) AS foodname,
fl_exclude,
"FL_unverified" AS fl_unverified
FROM clean.food_lookup
),
used AS (
SELECT 'seq1' AS slot,
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
UPPER(BTRIM(fb_fl_local_food_name)) AS foodname
FROM clean.food_bout
UNION ALL
SELECT 'seq2',
fb_fol_date,
fb_fol_b_focal_animid,
fb_begin_feed_time,
fb_end_feed_time,
UPPER(BTRIM(fb_local_food_name2))
FROM clean.food_bout
WHERE NULLIF(BTRIM(fb_local_food_name2), '') IS NOT NULL
)
SELECT used.*,
normalized_lookup.fl_exclude,
normalized_lookup.fl_unverified
FROM used
JOIN normalized_lookup USING (foodname)
WHERE normalized_lookup.fl_exclude IS TRUE
OR normalized_lookup.fl_unverified IS TRUE
ORDER BY used.fb_fol_date,
used.fb_fol_b_focal_animid,
used.fb_begin_feed_time,
used.slot;
</pre>

The 2026-09-13 snapshot had 18 excluded lookup rows and 359 unverified lookup
rows. Excluded names were referenced by 538 primary and 24 secondary bout
items, 562 occurrences total. Unverified names were referenced by 728 bout
items. These sets may overlap and must be profiled separately.

=== Solution ===

<code>FL_exclude = true</code> disqualifies the lookup entry. Do not load that
name into <code>FOOD_NAMES</code>, and exactly exclude every referring food-bout
item after approved case and edge-whitespace normalization.

<code>FL_unverified = true</code> does not disqualify a lookup entry or
referring bout. Preserve the flag in <code>clean</code>, document that it has no
production destination, and load the otherwise eligible name and bout.

The sanity gate must prove that no excluded name enters production and that no
unverified name is omitted merely because it is unverified.

== (#160) Converted food events require a consumer ROLES row ==

=== Problem ===

The mung food loader inserted <code>EVENTS</code> and
<code>FOOD_EVENTS</code> rows but inserted no <code>ROLES</code> row. Production
documentation says the food event's role identifies the individual consuming
the food. The roles trigger permits at most one role and requires that
participant to equal the related watch focal. Although absence currently
produces a warning rather than a hard constraint failure, omitting the consumer
would make the conversion incomplete.

=== Bad data ===

''Run after a rollback-only or disposable-database load.''

<pre>
SELECT events.eid,
watches.wid,
watches.animid AS focal,
COUNT(roles.pid) AS role_rows,
MIN(roles.role) AS role,
MIN(roles.participant) AS participant
FROM sokwedb.events
JOIN sokwedb.watches
ON watches.wid = events.wid
LEFT JOIN sokwedb.roles
ON roles.eid = events.eid
WHERE events.behavior = 'FOOD'
GROUP BY events.eid,
watches.wid,
watches.animid
HAVING COUNT(roles.pid) <> 1
OR MIN(roles.role) <> 'Solo'
OR MIN(roles.participant) <> watches.animid
ORDER BY events.eid;
</pre>

=== Solution ===

For every eligible food bout, insert exactly one <code>ROLES</code> row related
to the newly created event. Use role code <code>Solo</code> and the selected B
watch's exact <code>AnimID</code> as Participant. Capture the generated EID;
do not infer event identity from sequence state or source ordering.

This decision excludes no source rows. The loader and parity checks must prove
one food event, one Solo consumer role, and one or two contiguous food-detail
rows per eligible source bout.

== * (#161) FOOD_VARIATIONS conversion is deferred ==

=== Problem ===

<code>clean.food_variations_lookup</code> has a corresponding production
<code>housekeeping.FOOD_VARIATIONS</code> table, but the mung branch never
loaded it. The current food-event conversion requires decisions for missing
and unresolved LocalName values that are independent of loading food bouts and
their two support tables.

=== Bad data ===

''This is a dated scope diagnostic, not an exclusion query.''

<pre>
SELECT COUNT(*) AS source_rows,
COUNT(*) FILTER (
WHERE NULLIF(BTRIM(fvl_food_spelling_variant), '') IS NULL
) AS missing_variants,
COUNT(*) FILTER (
WHERE NULLIF(BTRIM(fvl_fl_local_food_name), '') IS NULL
) AS missing_local_names
FROM clean.food_variations_lookup;
</pre>

The 2026-09-13 snapshot contained 1,575 rows, no blank variants, 224 blank or
NULL local names, and 38 additional nonblank local names without a normalized
match in <code>clean.food_lookup</code>. Rerun and extend the query in the
future variations conversion; do not reuse these counts as exclusions.

=== Solution ===

Explicitly defer <code>clean.food_variations_lookup</code> to a separate future
conversion. The present food conversion must not insert
<code>FOOD_VARIATIONS</code> rows, must not depend on that table to normalize
bout names, and must not claim variation parity.

Preserve this issue and the source table for the future conversion. The
deferral excludes no food-bout rows and does not authorize dropping variation
data.

== (#162) Eligible FOOD_LOOKUP rows lack generalized scientific names ==

=== Problem ===

The approved <code>FOOD_NAMES.Description</code> source is
<code>fl_sci_food_name_gen</code>, but 360 eligible lookup rows have a blank or
NULL value. All 360 names are used by food bouts, in 812 primary and 111
secondary items. Of these names, 359 are marked <code>FL_unverified</code> and
must not be disqualified under Problem #159. The target description is NOT
NULL, nonempty, and case-insensitively unique.

=== Solution ===

Approved by the investigator on 2026-09-13: use
<code>Unknown -- <local name></code> for an eligible blank generalized name.
This stable value identifies the exact source name and does not consult
<code>fl_sci_food_name</code>. Nonblank descriptions continue to follow
Problem #87. This mapping excludes no rows.

== (#163) FOOD_BOUT duration disagrees with elapsed event time ==

=== Problem ===

The refreshed source has 1,136 rows where <code>fb_duration</code> differs from
the exact minutes between begin and end. Production stores the original begin
and end times but has no independent duration column.

=== Solution ===

Do not alter event times to reproduce <code>fb_duration</code>. Retain the
field in <code>clean.food_bout</code>, emit a sanity warning with the refreshed
count, and document that it has no production destination. This issue causes
no exclusions.

== (#164) FOOD_BOUT update metadata has no production destination ==

=== Problem ===

The refreshed source has 17,702 rows with non-NULL <code>fb_update</code>.
None of the food target tables has a corresponding update-metadata column.

=== Solution ===

Retain <code>fb_update</code> in <code>clean.food_bout</code>, emit a sanity
warning with the refreshed count, and do not overload event notes or another
production field. This issue causes no exclusions.

== * (#165) ATTENDANCE contains exact duplicate rows ==

=== Problem ===

The source table has no primary key and contains values that are identical in
all 17 columns. Loading every copy would create indistinguishable production
events and repeated sequence values. Retaining an arbitrary copy would not be
an exact source-row policy because no source value distinguishes the copies.

=== Bad data ===

''Run against the refreshed <code>clean</code> schema. The grouping explicitly
compares all 17 typed source columns.''

<pre>
SELECT attendance.*,
COUNT(*) AS copies
FROM clean.attendance AS attendance
GROUP BY attendance.a_date,
attendance.a_cl_community_id,
attendance.a_b_animid,
attendance.a_seq_num,
attendance.a_type_of_cycle,
attendance.a_observer1_id,
attendance.a_observer2_id,
attendance.a_bananas_given,
attendance.a_degree_of_arrival,
attendance.a_degree_of_departure,
attendance.a_time_start,
attendance.a_time_end,
attendance.a_duration_of_obs,
attendance.day,
attendance.mo,
attendance.yr,
attendance.cycle_old
HAVING COUNT(*) > 1
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num,
attendance.a_time_start;
</pre>

The 2026-09-14 snapshot contained 207 duplicate values, each occurring twice:
414 source rows total.

=== Solution ===

Approved by the investigator on 2026-09-14: exclude every copy of each exact
duplicate value. Do not use <code>DISTINCT</code> to retain one arbitrary copy.
The shared attendance projection and loader must include this query beside the
Problem #165 predicate and prove that every row whose complete typed value has
multiplicity greater than one is excluded.

Implemented in <code>conversion/attendance_projection.sql</code> as
<code>duplicate_count > 1</code>, partitioned over all 17 columns. On
2026-09-15 the investigator approved semantic validation: the dated 207 values
and 414 rows are audit evidence, not frozen acceptance criteria. Sanity
asserts that every refreshed duplicate copy is classified and excluded.
Remove only when exact duplicates leave the source.

== * (#166) ATTENDANCE sequence values are not contiguous per animal and date ==

=== Problem ===

Production attendance sequence values are expected to be exactly
<code>1..N</code> for each animal/date. Exact duplicates and other targeted
row exclusions can themselves create duplicate values or gaps, so sequence
validity must be checked after all other attendance exclusions.

=== Bad data ===

''This query expresses the approved staging against the 2026-09-14 schema.
Keep it synchronized with Problems #165 and #168 through #178 when predicates
change.''

<pre>
WITH source AS (
SELECT row_number() OVER () AS source_id,
attendance.*,
COUNT(*) OVER (
PARTITION BY attendance.a_date,
attendance.a_cl_community_id,
attendance.a_b_animid,
attendance.a_seq_num,
attendance.a_type_of_cycle,
attendance.a_observer1_id,
attendance.a_observer2_id,
attendance.a_bananas_given,
attendance.a_degree_of_arrival,
attendance.a_degree_of_departure,
attendance.a_time_start,
attendance.a_time_end,
attendance.a_duration_of_obs,
attendance.day,
attendance.mo,
attendance.yr,
attendance.cycle_old
) AS duplicate_count
FROM clean.attendance AS attendance
),
independent_eligible AS (
SELECT source.*
FROM source
JOIN sokwedb.biography_data AS biography
ON biography.animid = source.a_b_animid::text
WHERE source.duplicate_count = 1
AND source.a_date BETWEEN biography.entrydate AND biography.departdate
AND COALESCE(BTRIM(source.a_observer1_id::text), 'NONE')
<> COALESCE(BTRIM(source.a_observer2_id::text), 'NONE')
AND BTRIM(source.a_type_of_cycle::text)
NOT IN ('-', '0.30', '0.33')
AND source.cycle_old IS NOT NULL
AND source.cycle_old::text = BTRIM(source.cycle_old::text)
AND (source.a_bananas_given IS NULL
OR source.a_bananas_given = TRUNC(source.a_bananas_given))
AND source.a_time_start IS NOT NULL
AND source.a_time_end IS NOT NULL
AND source.a_time_start <= source.a_time_end
AND source.a_duration_of_obs IS NOT DISTINCT FROM
EXTRACT(EPOCH FROM
(source.a_time_end - source.a_time_start)) / 60
AND source.day IS NOT DISTINCT FROM
EXTRACT(DAY FROM source.a_date)::integer
AND source.mo IS NOT DISTINCT FROM
EXTRACT(MONTH FROM source.a_date)::integer
AND source.yr IS NOT DISTINCT FROM
EXTRACT(YEAR FROM source.a_date)::integer
),
overlap_rows AS (
SELECT DISTINCT first_row.source_id
FROM independent_eligible AS first_row
JOIN independent_eligible AS second_row
ON second_row.source_id <> first_row.source_id
AND second_row.a_date = first_row.a_date
AND second_row.a_b_animid = first_row.a_b_animid
AND second_row.a_time_start <= first_row.a_time_end
AND second_row.a_time_end >= first_row.a_time_start
),
eligible_before_seq AS (
SELECT independent_eligible.*
FROM independent_eligible
WHERE NOT EXISTS (
SELECT 1
FROM overlap_rows
WHERE overlap_rows.source_id = independent_eligible.source_id)
),
bad_groups AS (
SELECT a_date,
a_b_animid
FROM eligible_before_seq
GROUP BY a_date,
a_b_animid
HAVING MIN(a_seq_num) IS NULL
OR MIN(a_seq_num) <> 1
OR MAX(a_seq_num) IS DISTINCT FROM COUNT(*)
OR COUNT(DISTINCT a_seq_num) <> COUNT(*)
)
SELECT eligible_before_seq.*
FROM eligible_before_seq
JOIN bad_groups
USING (a_date, a_b_animid)
ORDER BY eligible_before_seq.a_date,
eligible_before_seq.a_b_animid,
eligible_before_seq.a_seq_num,
eligible_before_seq.a_time_start;
</pre>

The refreshed projection contained 453 invalid animal/date groups and 910 rows
after the other approved exclusions and the Problem #171 mapping.

=== Solution ===

Approved by the investigator on 2026-09-14: exclude every row in each invalid
animal/date group. Do not renumber, retain a partial group, or select rows by
order. Compute Problem #166 last, after Problems #165, #168 through #170, and
#172 through #178,
and include the complete identification query beside the loader predicate.
Sanity must prove every remaining group has non-NULL sequence values exactly
equal to <code>1..N</code>.

Implemented last in <code>conversion/attendance_projection.sql</code> by
<code>attendance_bad_seq_groups</code> and <code>problem_166</code>. The
dated 453 groups and 910 rows are audit evidence. Sanity proves every
refreshed invalid group is completely excluded and every eligible group is
exactly <code>1..N</code>. Remove only when all post-exclusion groups are
contiguous.

== (#167) ATTENDANCE observer values require PEOPLE support rows ==

=== Problem ===

The observer columns contain hundreds of case and edge-whitespace variants.
Observer codes must be matched by their standardized value so variants such as
<code>Mike</code>, <code>mike</code>, and <code>Mike </code> resolve to one
<code>PEOPLE.Person</code>. Standardized values absent from
<code>PEOPLE</code> require support rows.

=== Bad data ===

<pre>
WITH raw_observers AS (
SELECT a_observer1_id::text AS raw_person
FROM clean.attendance
WHERE a_observer1_id IS NOT NULL
UNION
SELECT a_observer2_id::text
FROM clean.attendance
WHERE a_observer2_id IS NOT NULL
), standardized AS (
SELECT raw_person,
LOWER(NORMALIZE(BTRIM(raw_person))) AS standard_person
FROM raw_observers
)
SELECT standardized.standard_person,
ARRAY_AGG(standardized.raw_person
ORDER BY standardized.raw_person) AS raw_spellings,
ARRAY_AGG(people.person ORDER BY people.person)
FILTER (WHERE people.person IS NOT NULL) AS existing_codes
FROM standardized
LEFT JOIN codes.people
ON LOWER(NORMALIZE(BTRIM(people.person))) =
standardized.standard_person
GROUP BY standardized.standard_person
ORDER BY standardized.standard_person;
</pre>

The refreshed 2026-09-14 snapshot had 932 distinct raw observer spellings in
718 standardized classes. Of those classes, 245 matched an existing
<code>PEOPLE</code> row and 473 required a new support row.

=== Solution ===

Approved by the investigator on 2026-09-14: standardize observer identifiers
with <code>LOWER(NORMALIZE(BTRIM(value)))</code>. Resolve a class to its existing
<code>PEOPLE.Person</code> spelling when present. For an unmatched class, choose
one deterministic trimmed source spelling, preferring mixed case and then
bytewise order, and add it as an active support row. Use that spelling for
Person and Name and <code>Attendance observer code: <code></code> for
Description. Map NULL to the existing <code>NONE</code> code. This support
decision excludes no rows except where Problem #168 also applies.

Implemented by <code>attendance_observer_codes</code> in
<code>conversion/attendance_projection.sql</code> and the ordered
<code>PEOPLE</code> insert in <code>conversion/load_attendance.sql</code>. Sanity
pins all 718 canonical codes using an explicitly byte-ordered, length-prefixed
serialization, all 932 raw-to-canonical mappings, and 473 missing support rows.
Remove only when every standardized observer value already resolves to one
active row.

== * (#168) ATTENDANCE Recorder and Observer2 resolve to the same person code ==

=== Problem ===

<code>ARRIVALS_A</code> requires Recorder and Observer2 to differ. Some source
rows contain observer spellings that become equal after standardizing case and
edge whitespace; rows with both observers NULL also become equal after the
approved NULL-to-<code>NONE</code> mapping.

=== Bad data ===

<pre>
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE LOWER(NORMALIZE(COALESCE(
BTRIM(attendance.a_observer1_id::text), 'NONE'))) =
LOWER(NORMALIZE(COALESCE(
BTRIM(attendance.a_observer2_id::text), 'NONE')))
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num,
attendance.a_time_start;
</pre>

The 2026-09-14 snapshot returned 1,544 rows: 1,484 with equal trimmed
non-NULL spellings, 39 additional case-equivalent pairs, and 21 with both
observers NULL.

=== Solution ===

Approved by the investigator on 2026-09-14: exactly exclude every returned
row. Do not fabricate or discard an observer. Include the query beside the
Problem #168 predicate and prove no eligible detail has equal observer codes.

Implemented as <code>problem_168</code> in
<code>conversion/attendance_projection.sql</code>. The dated 1,544 rows are
audit evidence; sanity proves that no refreshed eligible row has equal derived
observer codes. Remove only when no derived Recorder equals Observer2.

== * (#169) ATTENDANCE focal does not resolve to BIOGRAPHY_DATA ==

=== Problem ===

An attendance watch and its Solo role require an exact production AnimID.
Thousands of source rows use values absent from <code>BIOGRAPHY_DATA</code>,
mostly <code>NYANI</code>. Case or whitespace repair is not approved.

=== Bad data ===

<pre>
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE NOT EXISTS (
SELECT 1
FROM sokwedb.biography_data AS biography
WHERE biography.animid = attendance.a_b_animid::text)
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num,
attendance.a_time_start;
</pre>

The exact-match query returned 6,877 rows on 2026-09-14.

=== Solution ===

Approved by the investigator on 2026-09-14: exactly exclude every returned
row. Do not trim, case-fold, or map an unmatched focal to a sentinel. Include
the query beside the Problem #169 predicate and prove every eligible focal
resolves exactly.

Implemented as <code>problem_169</code> in
<code>conversion/attendance_projection.sql</code>. The dated 6,877 rows are
audit evidence; sanity proves that every refreshed eligible focal resolves.
Remove only when every exact source focal resolves.

== * (#170) ATTENDANCE focal is outside its study dates ==

=== Problem ===

A role participant must be under study on the event date. Some exact-matching
focals have attendance dates before EntryDate or after DepartDate.

=== Bad data ===

<pre>
SELECT attendance.*,
biography.entrydate,
biography.departdate
FROM clean.attendance AS attendance
JOIN sokwedb.biography_data AS biography
ON biography.animid = attendance.a_b_animid::text
WHERE attendance.a_date NOT BETWEEN
biography.entrydate AND biography.departdate
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
</pre>

The 2026-09-14 snapshot returned six rows.

=== Solution ===

Approved by the investigator on 2026-09-14: exactly exclude every returned
row. Include the query beside the Problem #170 predicate and prove every
eligible participant is under study on the event date.

Implemented as <code>problem_170</code> in
<code>conversion/attendance_projection.sql</code>. The dated six rows are audit
evidence; sanity proves that every refreshed eligible focal is under study on
its attendance date. Remove only when every exact focal is under study.

== (#171) ATTENDANCE direction fields contain the -9 sentinel ==

=== Problem ===

Production degrees may be NULL or integers from 0 through 359. The source uses
<code>-9</code> in one or both direction fields to mean that the direction was
not seen.

=== Bad data ===

<pre>
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE COALESCE(attendance.a_degree_of_arrival = -9, FALSE)
OR COALESCE(attendance.a_degree_of_departure = -9, FALSE)
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
</pre>

The 2026-09-14 snapshot returned 7,228 rows. Individual arrival and departure
counts overlap and must not be summed.

=== Solution ===

Confirmed by the project investigators on 2026-09-15: translate each
<code>-9</code> independently to NULL because it means the direction was not
seen. Preserve existing NULL and values from 0 through 359 exactly. Keep this
query beside the Problem #171 predicate to profile the mapped source condition,
but do not add Problem #171 to the exclusion array. Prove every eligible
target degree equals <code>NULLIF(source_degree, -9)</code> and satisfies the
target domain.

Implemented as the profiled <code>problem_171</code> condition and the
<code>NULLIF(..., -9)</code> arrival/departure projection in
<code>conversion/attendance_projection.sql</code>. The dated 7,228 rows are
audit evidence; no rows are excluded solely by this issue. Sanity proves each
refreshed eligible target equals <code>NULLIF(source_degree, -9)</code>. Remove
the mapping only when neither source degree contains <code>-9</code>.

== * (#172) ATTENDANCE swelling values have no approved CYCLE_STATES mapping ==

=== Problem ===

Most source swelling spellings have an exact lossless representation in
<code>CYCLE_STATES</code>. The values <code>-</code>, <code>0.30</code>, and
<code>0.33</code> do not. Mapping <code>-</code> to <code>MISS</code> or rounding
the numeric values would change source meaning.

=== Bad data ===

<pre>
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE BTRIM(attendance.a_type_of_cycle::text)
IN ('-', '0.30', '0.33')
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
</pre>

The 2026-09-14 snapshot returned 3,238 rows: 3,221 with <code>-</code>, two with
<code>0.30</code>, and 15 with <code>0.33</code>.

=== Solution ===

Approved by the investigator on 2026-09-14: exactly exclude every returned
row. Do not map or round these values. The explicit supported projection is:
<code>0</code>/<code>0.00</code> to <code>0</code>;
<code>.25</code>/<code>0.25</code> to <code>0.25</code>;
<code>.5</code>/<code>0.5</code>/<code>0.50</code> to <code>0.5</code>;
<code>.75</code>/<code>0.75</code> to <code>0.75</code>;
<code>1</code>/<code>1.00</code> to <code>1</code>;
<code>n/a</code>/<code>N/A</code> to <code>n/a</code>; and
<code>u</code>/<code>U</code> to <code>U</code>. Include the query beside the
Problem #172 predicate and reject any refreshed spelling outside this closed
mapping.

Implemented as <code>problem_172</code> in
<code>conversion/attendance_projection.sql</code>. The dated 3,238 rows are
audit evidence; sanity rejects any refreshed eligible spelling outside the
closed exact mapping. Remove only when every source spelling belongs to it.

== * (#173) ATTENDANCE CycleOld is missing or has edge spaces ==

=== Problem ===

<code>ARRIVALS_A.CycleOld</code> is required and may not contain only spaces.
The source has NULL values and edge-spaced values. Trimming would alter the
initial digitization recorded by this field.

=== Bad data ===

<pre>
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE attendance.cycle_old IS NULL
OR attendance.cycle_old::text <>
BTRIM(attendance.cycle_old::text)
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
</pre>

The 2026-09-14 snapshot returned 417 rows: 229 NULL and 188 edge-spaced values.

=== Solution ===

Approved by the investigator on 2026-09-14: exactly exclude every returned
row. Preserve all eligible CycleOld text exactly; do not trim or invent a
missing value. Include the query beside the Problem #173 predicate.

Implemented as <code>problem_173</code> in
<code>conversion/attendance_projection.sql</code>. The dated 417 rows are audit
evidence; sanity proves every refreshed eligible CycleOld is non-NULL and
edge-space free. Remove only when every source value satisfies that rule.

== * (#174) ATTENDANCE bananas contains a fractional value ==

=== Problem ===

<code>ARRIVALS_A.Bananas</code> is an integer. One source value is fractional,
and rounding or truncating it would alter the recorded value.

=== Bad data ===

<pre>
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE attendance.a_bananas_given IS NOT NULL
AND attendance.a_bananas_given <>
TRUNC(attendance.a_bananas_given)
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
</pre>

The 2026-09-14 snapshot returned one row with value <code>0.25</code>.

=== Solution ===

Approved by the investigator on 2026-09-14: exactly exclude the returned row.
Do not round or truncate it. Include the query beside the Problem #174
predicate and prove all eligible banana values are integral and in range.

Implemented as <code>problem_174</code> in
<code>conversion/attendance_projection.sql</code>. The dated one-row result is
audit evidence; sanity proves all refreshed eligible banana values are
integral and in range. Remove only when all source values are integral.

== * (#175) ATTENDANCE event interval is missing or inverted ==

=== Problem ===

Attendance event Start and Stop are required, and Start may not be after Stop.
One source row has no Start and other rows have Start after Stop. Production
attendance events cannot represent an inferred cross-midnight interval.

=== Bad data ===

<pre>
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE attendance.a_time_start IS NULL
OR attendance.a_time_end IS NULL
OR attendance.a_time_start > attendance.a_time_end
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
</pre>

The 2026-09-14 snapshot returned 28 rows: one missing Start, no missing Stop,
and 27 inverted intervals.

=== Solution ===

Approved by the investigator on 2026-09-14: exactly exclude every returned
row. Do not swap times, infer dates, derive Start from duration, or clamp a
value. Include the query beside the Problem #175 predicate.

Implemented as <code>problem_175</code> in
<code>conversion/attendance_projection.sql</code>. The dated 28 rows are audit
evidence; sanity proves every refreshed eligible interval has both endpoints
and is ordered. Remove only when all source intervals satisfy that rule.

== * (#176) ATTENDANCE duration disagrees with the event interval ==

=== Problem ===

For otherwise ordered intervals, some stored durations differ from the exact
minutes between Start and Stop. Production has no independent attendance
duration destination, so loading only the times would discard a conflicting
source assertion.

=== Bad data ===

<pre>
SELECT attendance.*,
EXTRACT(EPOCH FROM
(attendance.a_time_end - attendance.a_time_start)) / 60
AS computed_minutes
FROM clean.attendance AS attendance
WHERE attendance.a_time_start IS NOT NULL
AND attendance.a_time_start <= attendance.a_time_end
AND attendance.a_duration_of_obs IS DISTINCT FROM
EXTRACT(EPOCH FROM
(attendance.a_time_end - attendance.a_time_start)) / 60
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
</pre>

The 2026-09-14 snapshot returned 15 rows, separate from Problem #175.

=== Solution ===

Approved by the investigator on 2026-09-14: exactly exclude every returned
row. Do not alter event times or discard the disagreement as warning-only.
Include the query beside the Problem #176 predicate and prove exact agreement
for every eligible row.

Implemented as <code>problem_176</code> in
<code>conversion/attendance_projection.sql</code>. The dated 15 rows are audit
evidence; sanity proves every refreshed eligible duration agrees exactly with
elapsed time. Remove only when every otherwise valid source row agrees.

== * (#177) ATTENDANCE intervals overlap for one animal and date ==

=== Problem ===

Production warns when one animal is recorded at the feeding station in
overlapping intervals. Exact duplicates and independently invalid rows must
be removed first so this issue identifies only otherwise eligible overlap
participants.

=== Bad data ===

''This query is intentionally complete. Keep its independent eligibility
predicates synchronized with Problems #165 and #168 through #176/#178.''

<pre>
WITH source AS (
SELECT row_number() OVER () AS source_id,
attendance.*,
COUNT(*) OVER (
PARTITION BY attendance.a_date,
attendance.a_cl_community_id,
attendance.a_b_animid,
attendance.a_seq_num,
attendance.a_type_of_cycle,
attendance.a_observer1_id,
attendance.a_observer2_id,
attendance.a_bananas_given,
attendance.a_degree_of_arrival,
attendance.a_degree_of_departure,
attendance.a_time_start,
attendance.a_time_end,
attendance.a_duration_of_obs,
attendance.day,
attendance.mo,
attendance.yr,
attendance.cycle_old
) AS duplicate_count
FROM clean.attendance AS attendance
),
independent_eligible AS (
SELECT source.*
FROM source
JOIN sokwedb.biography_data AS biography
ON biography.animid = source.a_b_animid::text
WHERE source.duplicate_count = 1
AND source.a_date BETWEEN biography.entrydate AND biography.departdate
AND COALESCE(BTRIM(source.a_observer1_id::text), 'NONE')
<> COALESCE(BTRIM(source.a_observer2_id::text), 'NONE')
AND BTRIM(source.a_type_of_cycle::text)
NOT IN ('-', '0.30', '0.33')
AND source.cycle_old IS NOT NULL
AND source.cycle_old::text = BTRIM(source.cycle_old::text)
AND (source.a_bananas_given IS NULL
OR source.a_bananas_given = TRUNC(source.a_bananas_given))
AND source.a_time_start IS NOT NULL
AND source.a_time_end IS NOT NULL
AND source.a_time_start <= source.a_time_end
AND source.a_duration_of_obs IS NOT DISTINCT FROM
EXTRACT(EPOCH FROM
(source.a_time_end - source.a_time_start)) / 60
AND source.day IS NOT DISTINCT FROM
EXTRACT(DAY FROM source.a_date)::integer
AND source.mo IS NOT DISTINCT FROM
EXTRACT(MONTH FROM source.a_date)::integer
AND source.yr IS NOT DISTINCT FROM
EXTRACT(YEAR FROM source.a_date)::integer
)
SELECT DISTINCT first_row.*
FROM independent_eligible AS first_row
JOIN independent_eligible AS second_row
ON second_row.source_id <> first_row.source_id
AND second_row.a_date = first_row.a_date
AND second_row.a_b_animid = first_row.a_b_animid
AND second_row.a_time_start <= first_row.a_time_end
AND second_row.a_time_end >= first_row.a_time_start
ORDER BY first_row.a_date,
first_row.a_b_animid,
first_row.a_time_start,
first_row.a_time_end;
</pre>

The refreshed projection returned 428 otherwise eligible source rows after
Problem #171 became a mapping rather than an exclusion.
The unfiltered source had 463 overlap pairs, a count that includes other issue
classes and is not the exclusion baseline.

=== Solution ===

Approved by the investigator on 2026-09-14: exclude every row participating in
an otherwise eligible overlap. Do not select one interval from a pair. The
shared projection must define an execution-local source-row identity, include
this complete query in code, and prove zero overlaps remain before Problem
#166 sequence closure.

Implemented after independent exclusions as <code>problem_177</code> in
<code>conversion/attendance_projection.sql</code>. The dated 428 rows are audit
evidence; sanity proves that no refreshed eligible intervals overlap. Remove
only when no otherwise eligible intervals overlap.

== * (#178) ATTENDANCE date components disagree with A_date ==

=== Problem ===

The redundant source Day, Mo, and Yr values should describe
<code>A_date</code>. Some day or month values disagree. Discarding the
redundant values would hide a source conflict.

=== Bad data ===

<pre>
SELECT attendance.*
FROM clean.attendance AS attendance
WHERE attendance.day IS DISTINCT FROM
EXTRACT(DAY FROM attendance.a_date)::integer
OR attendance.mo IS DISTINCT FROM
EXTRACT(MONTH FROM attendance.a_date)::integer
OR attendance.yr IS DISTINCT FROM
EXTRACT(YEAR FROM attendance.a_date)::integer
ORDER BY attendance.a_date,
attendance.a_b_animid,
attendance.a_seq_num;
</pre>

The 2026-09-14 snapshot returned 79 distinct rows. Day disagreed in 46 rows,
month in 33, and year in none.

=== Solution ===

Approved by the investigator on 2026-09-14: exactly exclude every returned
row. Do not silently prefer one component representation. Include the query
beside the Problem #178 predicate and prove all eligible redundant components
agree with <code>A_date</code>.

Implemented as <code>problem_178</code> in
<code>conversion/attendance_projection.sql</code>. The dated 79 rows are audit
evidence; sanity proves every refreshed eligible redundant date component
agrees with <code>A_date</code>. Remove only when every source row agrees.

Problems #165, #166, #168 through #170, and #172 through #178 overlap and must
not be summed. Problem #171 is profiled but mapped, not excluded. In the
approved dependency order -- #165, independent #168-#170/#172-#176/#178,
#177, then #166 -- the refreshed union excluded 13,714 rows and left 120,385
eligible rows. Refresh each issue, the overlap set, the final sequence closure,
and the union before implementation.

On 2026-09-15 the investigator approved replacing complete-row fingerprints
and frozen issue-membership hashes with semantic validation. The documented
predicates and production contracts now determine acceptance. The conversion
uses an execution-local <code>source_id</code> to preserve row multiplicity and
prove relational parity; it is not a durable source identity. Historical
counts remain dated audit evidence and must be refreshed with
<code>attendance_profile</code>, including rebuilding the handoff Encounter Log
when testing a refreshed source.

== (#179) ATTENDANCE observers require standardized PEOPLE matching ==

=== Problem ===

Exact matching would create separate <code>PEOPLE</code> rows for case and
edge-whitespace variants of the same observer identifier. This contradicts
the case-equivalent unique index and the established conversion convention.

=== Bad Data ===

Run against the refreshed <code>clean</code>, <code>codes</code> snapshot after
food support loading:

<pre>
WITH raw_observers AS (
SELECT a_observer1_id::text AS raw_person
FROM clean.attendance
WHERE a_observer1_id IS NOT NULL
UNION
SELECT a_observer2_id::text
FROM clean.attendance
WHERE a_observer2_id IS NOT NULL
), standardized AS (
SELECT raw_person,
LOWER(NORMALIZE(BTRIM(raw_person))) AS standard_person
FROM raw_observers
)
SELECT standardized.standard_person,
ARRAY_AGG(standardized.raw_person
ORDER BY standardized.raw_person) AS raw_spellings,
ARRAY_AGG(people.person ORDER BY people.person)
FILTER (WHERE people.person IS NOT NULL) AS existing_codes
FROM standardized
LEFT JOIN codes.people
ON LOWER(NORMALIZE(BTRIM(people.person))) =
standardized.standard_person
GROUP BY standardized.standard_person
ORDER BY standardized.standard_person;
</pre>

The complete refreshed profile contained 932 raw spellings in 718 standardized
classes. Sixteen spellings had edge whitespace and 305 raw spellings mapped
to a different canonical code after trimming and case-equivalent resolution.

=== Solution ===

Approved by the investigator on 2026-09-14: retain the case-equivalent
<code>people_person_uniquenocase</code> index and the edge-whitespace constraint.
Map every raw attendance spelling through the standardized key defined in
Problem #167. Existing <code>PEOPLE.Person</code> spellings take precedence;
each previously unseen standardized class receives one deterministic support
code. Sanity must prove every raw spelling maps exactly once and every
standardized class produces exactly one canonical code. This issue excludes
no rows except the newly recognized standardized observer equalities under
Problem #168.

== (#180) load_finish deactivates ATTENDANCE observer codes ==

=== Problem ===

The existing finalization step deactivates <code>NONE</code> and every
<code>PEOPLE.Person</code> containing a slash. After attendance loading this
makes three referenced observer codes inactive: <code>NONE</code>,
<code>N/A</code>, and <code>SELEMANI/YAHAYA</code>. They account for 37,655
Recorder or Observer2 uses, violating the requirement that attendance observer
codes remain active.

=== Bad Data ===

Run after attendance and <code>load_finish</code>:

<pre>
SELECT people.person,
COUNT(*) AS observer_uses
FROM (
SELECT arrivals_a.recorder AS person FROM sokwedb.arrivals_a
UNION ALL
SELECT arrivals_a.observer2 FROM sokwedb.arrivals_a
) AS attendance_observers
JOIN codes.people USING (person)
WHERE NOT people.active
GROUP BY people.person
ORDER BY people.person;
</pre>

The refreshed query returned three codes and 37,655 observer uses:
<code>N/A</code> 54 times, <code>NONE</code> 37,599 times, and
<code>SELEMANI/YAHAYA</code> twice.

=== Solution ===

Approved by the investigator on 2026-09-14 as part of the active raw-observer
policy: <code>conversion/load_finish.sql</code> retains active status for any
person referenced by <code>ARRIVALS_A.Recorder</code> or
<code>ARRIVALS_A.Observer2</code>. Problem #39 finalization remains unchanged
for unreferenced <code>UNK</code>, <code>NONE</code>, and slash-containing codes.
This issue excludes no attendance rows. Remove the exception only when no
attendance detail references a code that finalization would deactivate.

== (#181) COLOBUS descriptive text fields contain NULL ==

=== Problem ===

The Access <code>COLOBUS</code> source contains NULL in eight descriptive text
fields whose production destinations are required text columns. NULL occurs
in <code>col_hunter_id</code> 722 times, <code>col_killer_id</code> 1,438 times,
<code>col_victim_age</code> 1,447 times, <code>col_colobus_details</code> 1,652 times,
<code>col_veg_details</code> 2,073 times, <code>col_hunt_failure_details</code> 1,943 times,
<code>col_comments</code> 885 times, and <code>col_hunt_description</code>
1,803 times.

The seven colobus-specific values map to <code>COLOBUS.Hunters</code>,
<code>COLOBUS.Killers</code>, <code>COLOBUS.VictimsAges</code>, <code>COLOBUS.Details</code>,
<code>COLOBUS.Vegetation</code>, <code>COLOBUS.FailureDetails</code>, and <code>COLOBUS.Description</code>.
Comments map separately to the shared <code>EVENTS.Notes</code> column.

=== Bad Data ===

<pre>
SELECT COUNT(*) FILTER (WHERE col_hunter_id IS NULL) AS hunters,
COUNT(*) FILTER (WHERE col_killer_id IS NULL) AS killers,
COUNT(*) FILTER (WHERE col_victim_age IS NULL) AS victims_ages,
COUNT(*) FILTER (WHERE col_colobus_details IS NULL) AS troop_details,
COUNT(*) FILTER (WHERE col_veg_details IS NULL) AS vegetation,
COUNT(*) FILTER (WHERE col_hunt_failure_details IS NULL)
AS failure_details,
COUNT(*) FILTER (WHERE col_comments IS NULL) AS comments,
COUNT(*) FILTER (WHERE col_hunt_description IS NULL) AS descriptions
FROM clean.colobus;
</pre>

The refreshed query returned, in column order, 722, 1,438, 1,447, 1,652,
2,073, 1,943, 885, and 1,803 rows.

=== Solution ===

Approved by the investigator on 2026-09-15: retain the existing production
schema and map each NULL independently to the empty string. Implement each
destination as <code>COALESCE(source_value, '')</code>; preserve every non-NULL
source value exactly. The existing required-text and not-only-spaces
constraints permit the empty string, so this normalization requires no schema
change. Do not concatenate comments with the encounter description, and do
not exclude a source row solely because any of these eight fields is NULL.

Sanity must prove exact field-by-field <code>COALESCE(source_value, '')</code>
parity and zero exclusions attributable only to Problem #181. Remove this
normalization only when all eight source fields contain no NULL values.

== (#182) MATING_EVENT community codes differ only by case ==

=== Problem ===

The Access <code>MATING_EVENT.M_cl_community_id</code> column contains the value <code>kk</code>,
which differs only by case from the existing production <code>COMM_IDS.CommID</code> code
<code>KK</code>. Production community codes are case-sensitive.

=== Bad Data ===

<pre>
SELECT *
FROM clean.mating_event
WHERE m_cl_community_id = 'kk';
</pre>

The refreshed source contains 181 such rows. All other mating community
values are <code>KK</code> or <code>MT</code>.

=== Solution ===

Approved by the investigators on 2026-09-15: standardize
<code>m_cl_community_id = 'kk'</code> to the existing <code>KK</code> code in the clean schema. This
is a case-only spelling normalization and excludes no rows. Preserve all
other source community values unchanged, and require the mating sanity check
to verify that every resulting community exists in <code>COMM_IDS</code>.

== (#183) MATING_EVENT times may contain seconds ==

=== Problem ===

Production <code>EVENTS.Start</code> and <code>EVENTS.Stop</code> record times to minute precision
and reject nonzero seconds. The Access <code>MATING_EVENT.M_time</code> value is initially
represented as a timestamp, and merely converting it to a PostgreSQL time
value does not remove seconds.

=== Bad Data ===

<pre>
SELECT *
FROM raw."MATING_EVENT"
WHERE extract(second FROM "M_time") <> 0;
</pre>

The refreshed source contains one row, dated 2008-10-12 with focal GA, male
WL, female GA, and time 09:35:30. Problem #4 remains corrected: the refreshed
source has no non-midnight time component in <code>M_FOL_date</code>.

=== Solution ===

Approved by the investigators on 2026-09-15: mating event times are recorded
to minute precision. During construction of the tidy schema, truncate
<code>M_time</code> to the minute before <code>tidy.sql</code> converts the column to <code>TIME</code>. Do not
round the value. The mating sanity check must reject the conversion if a
value containing seconds nevertheless reaches the clean schema. This issue
excludes no rows and does not resolve missing or out-of-window mating times.

== * (#184) MATING_EVENT ordinary boolean flags contain question marks ==

=== Problem ===

Production <code>MATINGS.Incest</code>, <code>Consort</code>, <code>Guarding</code>, <code>Courting</code>, and <code>Camp</code> are
required Boolean values. The Access fields use <code>X</code> for true and blank or NULL
for false, but one <code>M_court_flag</code> row and two different <code>M_camp_flag</code> rows
contain <code>?</code>. The unknown values cannot be represented by required Booleans.

=== Bad Data ===

<pre>
SELECT *
FROM clean.mating_event
WHERE BTRIM(m_court_flag) = '?'
OR BTRIM(m_camp_flag) = '?';
</pre>

The refreshed source contains three distinct rows. No row contains <code>?</code> in
both fields, and none overlaps the Problem #186 exclusion.

=== Solution ===

Approved by the investigators on 2026-09-15: map <code>X</code> to true and blank or NULL
to false for all five fields. Exclude the three rows containing <code>?</code>. Sanity
must reject any other domain value. This issue independently excludes three
rows and contributes three rows to the exclusion union.

== (#185) MATING_EVENT records two interference facts in one target field ==

=== Problem ===

<code>M_interference</code> records a description or chimp ID.
<code>M_interference_too_late_flag</code> records the ID of a chimp who interfered too
late. Production has only the single required text column
<code>MATINGS.Interference</code>, so copying or concatenating the fields without labels
would lose which fact each value represents.

=== Bad Data ===

<pre>
SELECT COUNT(*) FILTER (
WHERE NULLIF(BTRIM(m_interference), '') IS NOT NULL
AND NULLIF(BTRIM(m_interference_too_late_flag), '') IS NOT NULL)
AS both_fields,
COUNT(*) FILTER (
WHERE NULLIF(BTRIM(m_interference), '') IS NOT NULL
AND NULLIF(BTRIM(m_interference_too_late_flag), '') IS NULL)
AS interference_only,
COUNT(*) FILTER (
WHERE NULLIF(BTRIM(m_interference), '') IS NULL
AND NULLIF(BTRIM(m_interference_too_late_flag), '') IS NOT NULL)
AS too_late_only
FROM clean.mating_event;
</pre>

The refreshed source contains 82 rows with both fields, 3,760 with only
<code>m_interference</code>, and 135 with only the too-late field. No source
<code>m_interference</code> contains the labels <code>Interference:</code> or <code>Too late:</code>.

=== Solution ===

Approved by the investigator on 2026-09-16: trim both fields and use this
explicit encoding in <code>MATINGS.Interference</code>:

* neither field: the empty string;
* <code>m_interference</code> only: the trimmed source value;
* too-late only: <code>Too late: </code> followed by the trimmed source value; and
* both fields: <code>Interference: </code> followed by the trimmed interference value,
then <code>; Too late: </code> followed by the trimmed too-late value.

This preserves the two source facts without changing the production schema.
Problem #186 separately excludes too-late <code>X</code> values before this encoding.
This issue excludes no rows.

== * (#186) MATING_EVENT too-late interference contains X instead of an ID ==

=== Problem ===

The investigators identified <code>M_interference_too_late_flag</code> as the ID of a
chimp who interfered too late, but 110 source rows contain <code>X</code>. <code>X</code> is not a
chimp ID, and the source does not identify an individual who can replace it.

=== Bad Data ===

<pre>
SELECT *
FROM clean.mating_event
WHERE BTRIM(m_interference_too_late_flag) = 'X';
</pre>

The 110 rows span 1976 through 2015 and all use source <code>B-REC</code>. Eighty-two
also contain a separate interference description or ID; 28 do not. In
contrast, the 107 chimp-like values occur only in 2012 through 2015 and always
have an empty main interference field. The evidence is consistent with <code>X</code>
being an older marker, but it does not recover the required chimp ID.

=== Solution ===

Approved by the investigator on 2026-09-16: exclude all 110 rows containing
<code>X</code> in <code>m_interference_too_late_flag</code>. Do not interpret <code>X</code> as an individual
or silently discard the too-late fact. This issue independently excludes 110
rows, overlaps none of the three Problem #184 rows, and increases the
exclusion union from 3 to 113 rows.

== * (#187) MATING_EVENT fail flags contain missing and unknown values ==

=== Problem ===

Production <code>MATINGS.Fail</code> is a required Boolean. The investigators confirmed
that <code>X</code>, <code>Y</code>, and <code>y</code> mean the mating was not complete and map to true, while
<code>N</code> maps to false. The Access source also contains blank, NULL, and <code>?</code>
values, which require an explicit disposition.

=== Bad Data ===

<pre>
SELECT COALESCE(NULLIF(BTRIM(m_fail_flag), ''), '[blank/NULL]') AS fail_value,
COUNT(*)
FROM clean.mating_event
GROUP BY COALESCE(NULLIF(BTRIM(m_fail_flag), ''), '[blank/NULL]')
ORDER BY fail_value;
</pre>

The refreshed source contains 21,786 blank or NULL values and 552 <code>?</code> values.
After Problems #184 and #186, 21,676 blank or NULL values and 551 <code>?</code> values
remain eligible.

=== Solution ===

Approved by the investigator on 2026-09-16: map <code>X</code>, <code>Y</code>, and <code>y</code> to true;
map <code>N</code>, blank, and NULL to false; and exclude rows containing <code>?</code>. Sanity
must reject any other domain value. The <code>?</code> condition independently excludes
552 rows, overlaps one Problem #184 row and no Problem #186 rows, and increases
the exclusion union from 113 to 664 rows. It contributes 551 newly excluded
rows.

== * (#188) MATING_EVENT times fall outside the permitted event window ==

=== Problem ===

Except for the separately governed <code>sdb_no_time</code> sentinel, production
<code>EVENTS.Start</code> cannot be before 04:00 and <code>EVENTS.Stop</code> cannot be after 20:00.
Some non-NULL Access mating times fall outside that interval.

=== Bad Data ===

<pre>
SELECT *
FROM clean.mating_event
WHERE m_time IS NOT NULL
AND (m_time < '04:00'::TIME OR m_time > '20:00'::TIME)
ORDER BY m_fol_date, m_time;
</pre>

The refreshed source contains 25 rows: 24 between 01:00 and 03:48 and one at
21:38. None overlaps Problems #184, #186, or #187.

=== Solution ===

Approved by the investigator on 2026-09-16: exclude the 25 rows with non-NULL
times outside 04:00 through 20:00. Do not clamp, wrap, or reinterpret their
times. This issue independently and newly excludes 25 rows, increasing the
exclusion union from 664 to 689 rows.

== (#189) MATING_EVENT rows may have no recorded time ==

=== Problem ===

The Access source contains 34 mating rows with NULL <code>m_time</code>. Production
<code>EVENTS.Start</code> and <code>EVENTS.Stop</code> are required. Before this decision, only
aggression, B-record-note, and pantgrunt events could use the established
<code>sdb_no_time</code> value to distinguish an unrecorded time from an observed time.

=== Bad Data ===

<pre>
SELECT *
FROM clean.mating_event
WHERE m_time IS NULL;
</pre>

The refreshed source contains 34 rows. One overlaps the prior exclusion
union, so excluding missing times would discard 33 additional otherwise
eligible mating records.

=== Solution ===

Approved by the investigator on 2026-09-16: permit mating events to use the
existing <code>sdb_no_time</code> sentinel. Add <code>sdb_mating_event</code> to the two owning
<code>EVENTS</code> constraint allowlists and document the expanded behavior set. At the
production load boundary, map only SQL NULL <code>m_time</code> values to <code>sdb_no_time</code>
for both <code>EVENTS.Start</code> and <code>EVENTS.Stop</code>; preserve every non-NULL source time.
The existing point-event and paired-sentinel constraints remain in force.
This issue excludes no rows.

== (#190) MATING_EVENT swelling values may be missing ==

=== Problem ===

Production <code>MATINGS.Swelling</code> is required, but the Access source contains 572
NULL values and 16 <code>?</code> values. The source context for the question marks
describes unavailable information, including "no estrous state noted" and
"swelling not recorded".

=== Bad Data ===

<pre>
SELECT *
FROM clean.mating_event
WHERE m_fem_swelling IS NULL
OR BTRIM(m_fem_swelling) = '?';
</pre>

After prior exclusions, 568 NULL values and all 16 question marks remain
eligible.

=== Solution ===

Approved by the investigator on 2026-09-16: map both NULL and <code>?</code> to the
existing <code>CYCLE_STATES</code> code <code>MISS</code>, documented as "Missing data". <code>MISS</code> has
NULL <code>AsNum</code> and <code>SSRank</code>, so these records do not contribute a measured value
to daily swelling minimum/maximum calculations. The original missing-value
encoding remains available in <code>clean</code>. This issue excludes no rows.

== (#191) MATING_EVENT records one-third swelling ==

=== Problem ===

Nine otherwise eligible Access mating rows contain the swelling value <code>0.33</code>,
which is absent from <code>CYCLE_STATES</code>. Inserting the value into <code>MATINGS</code>
reproduces a <code>matings_swelling_fkey</code> violation.

=== Bad Data ===

<pre>
SELECT *
FROM clean.mating_event
WHERE BTRIM(m_fem_swelling) = '0.33';
</pre>

=== Solution ===

Approved by the investigator on 2026-09-16: add <code>CYCLE_STATES</code> code <code>0.33</code>
with <code>AsNum</code> 0.33, <code>SSRank</code> 3, and description "1/3 swollen". Shift the
existing <code>0.5</code>, <code>0.75</code>, <code>1</code>, and <code>U</code> support rows to ranks 4 through 7 so the
unique integer rank continues to order measured swelling values correctly.
Map source <code>0.33</code> to the new exact code. This issue excludes no rows.

== (#192) MATING_EVENT extractors require PEOPLE support ==

=== Problem ===

Production <code>MATINGS.ExtractedBy</code> is required and must reference an active
<code>PEOPLE.Person</code>. The source contains 20,747 NULL extractor values, one
case-only spelling difference, and three populated names absent from the
conversion <code>PEOPLE</code> table.

=== Bad Data ===

<pre>
SELECT m_extracted_by, COUNT(*)
FROM clean.mating_event
WHERE NULLIF(BTRIM(m_extracted_by), '') IS NULL
OR NOT EXISTS (
SELECT 1
FROM clean.people
WHERE LOWER(NORMALIZE(BTRIM(people.person))) =
LOWER(NORMALIZE(BTRIM(mating_event.m_extracted_by))))
GROUP BY m_extracted_by
ORDER BY m_extracted_by;
</pre>

<code>Karen McLellan</code> matches the existing stored spelling <code>KAREN MCLELLAN</code>.
<code>Deus Mjungu</code>, <code>FELDBLUM</code>, and <code>Joclyn Antonio</code> have no standardized match.

=== Solution ===

Approved by the investigator on 2026-09-16: map a NULL, empty, or
whitespace-only extractor to <code>UNK</code>; trim populated values; reuse the exact
stored <code>PEOPLE.Person</code> spelling for standardized case-equivalent matches; and
add <code>Deus Mjungu</code>, <code>FELDBLUM</code>, and <code>Joclyn Antonio</code> as active support people.
Use <code>LOWER(NORMALIZE(BTRIM(value)))</code> as the deterministic matching key. This
issue excludes no rows.

== (#193) MATING_EVENT comments require a production destination ==

=== Problem ===

The Access source stores mating comments separately from the full mating
description. Production has one shared event-notes field, <code>EVENTS.Notes</code>,
and silently concatenating the two source fields would erase that distinction.
The source contains 14,294 NULL comments and 15,134 comments that are NULL or
empty after trimming.

=== Bad Data ===

<pre>
SELECT COUNT(*) FILTER (WHERE m_comments IS NULL) AS null_comments,
COUNT(*) FILTER (
WHERE NULLIF(BTRIM(m_comments), '') IS NULL) AS blank_comments
FROM clean.mating_event;
</pre>

=== Solution ===

Approved by the investigator on 2026-09-16: map <code>m_comments</code> alone to
<code>EVENTS.Notes</code>. Map SQL NULL to the empty string with
<code>COALESCE(m_comments, '')</code> and preserve every non-NULL source value exactly.
Do not concatenate <code>m_full_description</code> into event notes. This issue excludes
no rows.

== (#194) MATING_EVENT full descriptions require a mating-specific destination ==

=== Problem ===

The Access source stores a full mating description in <code>m_full_description</code>.
<code>BRECORD_NOTES</code> cannot preserve it because that detail table can attach only
to a B-record event, cannot share the mating EID, and requires unrelated
B-record text fields. The source contains 18,212 descriptions that are NULL
or empty after trimming.

=== Bad Data ===

<pre>
SELECT COUNT(*) FILTER (WHERE m_full_description IS NULL)
AS null_descriptions,
COUNT(*) FILTER (
WHERE NULLIF(BTRIM(m_full_description), '') IS NULL)
AS blank_descriptions
FROM clean.mating_event;
</pre>

=== Solution ===

Approved by the investigator on 2026-09-16: add the required text column
<code>MATINGS.Description</code> and map <code>m_full_description</code> to it. Map SQL NULL to the
empty string with <code>COALESCE(m_full_description, '')</code> and preserve every
non-NULL source value exactly. The production constraint permits the empty
string but rejects nonempty whitespace-only values.

The permanent schema and documentation change was implemented as append-only
master commit <code>a628fdd</code> (<code>feat(matings): preserve mating descriptions</code>) and
fast-forwarded to local, origin, private, and blessed master. This issue
excludes no rows.

== (#195) MATING_EVENT sources require MATING_SOURCES support ==

=== Problem ===

Production <code>MATINGS.Source</code> is required and references
<code>MATING_SOURCES.Source</code>, but the source code table is empty. The Access source
contains mixed-case source labels while the production support key requires
uppercase values.

=== Bad Data ===

<pre>
SELECT m_source, COUNT(*) AS row_count,
MIN(m_fol_date) AS first_date,
MAX(m_fol_date) AS last_date
FROM clean.mating_event
GROUP BY m_source
ORDER BY m_source;
</pre>

The refreshed source contains <code>B-REC</code> 26,066 times over 1976 through 2018,
<code>Tiki</code> 393 times over 2000 through 2007, and <code>Bobu</code> 24 times in 2000. The
repository provides no more authoritative definition of <code>Bobu</code> than its
source label.

=== Solution ===

Approved by the investigator on 2026-09-16: add literal support codes <code>B-REC</code>,
<code>TIKI</code>, and <code>BOBU</code>, described respectively as "Mating record sourced from
B-REC", "Mating record sourced from Tiki", and "Mating record sourced from
Bobu". At the production boundary map source <code>B-REC</code> to <code>B-REC</code>, <code>Tiki</code> to
<code>TIKI</code>, and <code>Bobu</code> to <code>BOBU</code>. Do not claim semantics beyond the source labels.
Sanity must reject any refreshed source value outside this approved domain and
verify that every transformed code exists exactly once. This issue excludes
no rows.

== * (#196) MATING_EVENT participants are absent from BIOGRAPHY_DATA ==

=== Problem ===

Production <code>ROLES.Participant</code> is required and references
<code>BIOGRAPHY_DATA.AnimID</code>. The first exclusion-free loader run failed while
inserting female participant <code>STR</code> for conversion identity 117. The source
also contains other participant values that are not production biography
identifiers, including category labels, unknown markers, blanks, and
unresolved spellings.

=== Bad Data ===

<pre>
WITH eligible AS (
SELECT *
FROM clean.mating_event
WHERE COALESCE(BTRIM(m_court_flag), '') <> '?'
AND COALESCE(BTRIM(m_camp_flag), '') <> '?'
AND COALESCE(BTRIM(m_interference_too_late_flag), '') <> 'X'
AND COALESCE(BTRIM(m_fail_flag), '') <> '?'
AND (m_time IS NULL
OR m_time BETWEEN '04:00'::TIME AND '20:00'::TIME)
)
SELECT DISTINCT eligible.m_conversion_id
FROM eligible
CROSS JOIN LATERAL (
VALUES (eligible.m_male), (eligible.m_female)
) AS participant(animid)
WHERE NOT EXISTS (
SELECT 1
FROM sokwedb.biography_data
WHERE biography_data.animid = BTRIM(participant.animid))
ORDER BY eligible.m_conversion_id;
</pre>

The refreshed post-follow database contains 574 missing participant
occurrences in 566 otherwise eligible source rows, representing 34 distinct
trimmed values. The first failure is identity 117, dated 2000-10-14, with
focal <code>BE</code>, male <code>SL</code>, and female <code>STR</code>. The failed first chunk rolled back
and left zero <code>MATE</code> events.

=== Solution ===

Approved by the investigator on 2026-09-16: exclude every source row for which
either trimmed participant is absent from <code>BIOGRAPHY_DATA</code>. Do not create
biography rows or substitute participant identities without source evidence.
Sanity must verify that this condition selects exactly 566 distinct eligible
source identities in the retained snapshot. The rows overlap none of the
prior 689-row exclusion union and increase it to 1,255 rows, leaving 25,228
rows eligible before later failures.

== * (#197) MATING_EVENT participants fall outside biography study dates ==

=== Problem ===

After excluding Problem #196 rows, the loader failed when actor <code>FAM</code>
participated on 2014-09-10, after the individual's 2012-04-26 departure date.
Production requires every role participant to be biography-valid on the event
date.

=== Bad Data ===

<pre>
SELECT DISTINCT mating_event.m_conversion_id
FROM clean.mating_event
CROSS JOIN LATERAL (
VALUES (mating_event.m_male), (mating_event.m_female)
) AS participant(animid)
JOIN sokwedb.biography_data
ON biography_data.animid = BTRIM(participant.animid)
WHERE mating_event.m_fol_date < biography_data.entrydate
OR mating_event.m_fol_date > biography_data.departdate
ORDER BY mating_event.m_conversion_id;
</pre>

After applying Problems #184, #186, #187, #188, and #196, 28 participant
occurrences in 28 source rows fall outside study dates. Three are male actors
after departure, 13 are female actees after departure, and 12 are female
actees before entry. The affected participants are <code>AT</code>, <code>CF</code>, <code>FAM</code>, <code>OR</code>,
<code>SF</code>, <code>SH</code>, <code>SIF</code>, <code>SP</code>, <code>WN</code>, and <code>YD</code>. The failed first chunk rolled back
and left zero <code>MATE</code> events.

=== Solution ===

Approved by the investigator on 2026-09-16: exclude all 28 rows. Do not alter
source dates or biography study spans. Sanity must verify the exact 28-row
condition after the prior exclusions. This increases the exclusion union
from 1,255 to 1,283 rows and leaves 25,200 rows eligible before later
failures.

== * (#198) MATING_EVENT focal values are absent from BIOGRAPHY_DATA ==

=== Problem ===

After Problems #196 and #197, the loader failed while creating an Other watch
for focal <code>LUT</code>, which is absent from <code>BIOGRAPHY_DATA</code>. Production requires
every <code>WATCHES.AnimID</code>, including mating-only Other watches, to be a biography
identifier or the established no-focal sentinel.

=== Bad Data ===

<pre>
SELECT COALESCE(NULLIF(BTRIM(m_fol_b_focal_animid), ''), 'NONE') AS focal,
COUNT(*) AS row_count
FROM clean.mating_event
WHERE NOT EXISTS (
SELECT 1
FROM sokwedb.biography_data
WHERE biography_data.animid = COALESCE(
NULLIF(BTRIM(mating_event.m_fol_b_focal_animid), ''), 'NONE'))
GROUP BY COALESCE(NULLIF(BTRIM(m_fol_b_focal_animid), ''), 'NONE')
ORDER BY focal;
</pre>

After the prior exclusions, 311 otherwise eligible rows use 51 unsupported
focal values. Three source rows have unique case-insensitive biography
matches: <code>Gb</code> to <code>GB</code>, <code>lam</code> to <code>LAM</code>, and <code>sif</code> to <code>SIF</code>. The remaining 308
rows use 48 unsupported values, many representing groups, multiple animals,
or missing-target markers. The failed first chunk rolled back and left zero
<code>MATE</code> events.

=== Solution ===

Approved by the investigator on 2026-09-16: normalize only the three proven
case variants in <code>clean</code>, then exclude the 308 rows whose normalized focal is
still absent from <code>BIOGRAPHY_DATA</code>. Do not infer individual focal identities
from group or multi-animal values. Sanity must verify the exact 308-row
residual after prior exclusions. This increases the exclusion union from
1,283 to 1,591 rows and leaves 24,892 rows eligible before later failures.

== * (#199) MATING_EVENT focal dates are outside BIOGRAPHY_DATA study dates ==

=== Problem ===

After Problem #198, the loader failed while creating an Other watch for focal
<code>POR</code> on 2016-09-22. <code>POR</code> departed on 2014-03-13, and production requires a
watch date to fall within the focal animal's biography study dates.

=== Bad Data ===

<pre>
SELECT mating_event.m_conversion_id,
mating_event.m_fol_date,
focal_biography.animid,
focal_biography.entrydate,
focal_biography.departdate
FROM clean.mating_event
JOIN sokwedb.biography_data AS focal_biography
ON focal_biography.animid = COALESCE(
NULLIF(BTRIM(mating_event.m_fol_b_focal_animid), ''), 'NONE')
WHERE mating_event.m_fol_date < focal_biography.entrydate
OR mating_event.m_fol_date > focal_biography.departdate
ORDER BY mating_event.m_conversion_id;
</pre>

After the prior exclusions, 43 otherwise eligible rows have focal dates
outside the focal's study dates. Forty-one rows use <code>POR</code> after departure;
two use <code>SN</code> before entry. The failure occurred after 16,000 rows had
committed in earlier chunks, while the failing chunk rolled back.

=== Solution ===

Approved by the investigator on 2026-09-16: exclude all 43 rows. Do not alter
source dates or biography study spans. Sanity must verify the exact 43-row
condition after the prior exclusions. This increases the exclusion union
from 1,591 to 1,634 rows and leaves 24,849 rows eligible before later
failures. Rebuild the post-follow baseline before retrying so committed MATE
rows and any mating-created Other watches are removed together.

== (#200) COLOBUS encounters have no recorded Stop ==

=== Problem ===

There are 204 COLOBUS source rows with a recorded Start but no Stop. The
general EVENTS sentinel rules did not permit preserving a known Start with an
unknown Stop for behavior <code>COL</code>.

=== Bad Data ===

<pre>
SELECT col_encounter_date, col_fol_b_focal_chimp_id, col_begin_time
FROM clean.colobus
WHERE col_end_time IS NULL
ORDER BY col_encounter_date, col_fol_b_focal_chimp_id, col_begin_time;
</pre>

=== Solution ===

Approved for this conversion: retain each recorded Start and map only the
missing Stop to <code>sdb_no_time</code>. Add a narrow, documented EVENTS exception for
COL with a non-sentinel Start and sentinel Stop. Do not exclude these rows or
allow a sentinel COL Start. Validation must find exactly 204 converted COL
events with the sentinel Stop.

== * (#201) COLOBUS encounter times violate the approved time range ==

=== Problem ===

Fifteen rows cannot be represented without changing a recorded event time:
eight have Start after Stop, two have Start after 20:00, and seven have Stop
after 20:00. The two late Starts overlap the Start-after-Stop class, so the
distinct union is 15 rows. None has a missing Stop.

=== Bad Data ===

<pre>
SELECT col_encounter_date, col_fol_b_focal_chimp_id,
col_begin_time, col_end_time
FROM clean.colobus
WHERE col_begin_time > col_end_time
OR col_begin_time > '20:00'::TIME
OR col_end_time > '20:00'::TIME
ORDER BY col_encounter_date, col_fol_b_focal_chimp_id, col_begin_time;
</pre>

=== Solution ===

Approved for this conversion: exclude exactly this 15-row union at the loader
boundary and preserve all other recorded times. Sanity must fail if the
predicate no longer identifies exactly 15 rows. This establishes an
exclusion union of 15 rows before Problem #210.

== (#202) COLOBUS StartMap is missing from seven rows ==

=== Problem ===

Seven source rows have no <code>COL_begin_time_MAP</code>. Of the 2,240 recorded values,
126 intentionally differ from the standard 15-minute calculation and must not
be recalculated.

=== Bad Data ===

<pre>
SELECT COUNT(*) FILTER (WHERE col_begin_time_map IS NULL) AS missing,
COUNT(*) FILTER (
WHERE col_begin_time_map IS NOT NULL
AND col_begin_time_map IS DISTINCT FROM
DATE_BIN('15 minutes',
DATE '1960-07-04' + col_begin_time
+ INTERVAL '8 minutes',
TIMESTAMP '1960-07-04 00:00')::TIME) AS adjusted
FROM clean.colobus;
</pre>

=== Solution ===

Approved for this conversion: derive only the seven missing values at the
production load boundary with the displayed <code>DATE_BIN</code> expression. Preserve
all recorded values exactly. This excludes no rows.

== (#203) COLOBUS Hunt, HuntCert, and Kill can be unknown ==

=== Problem ===

The source contains 15 NULL Hunt flags, 16 NULL HuntCert flags, and 221 NULL
Kill flags, but the corresponding production columns were <code>NOT NULL</code>.

=== Bad Data ===

<pre>
SELECT COUNT(*) FILTER (WHERE col_hunt_flag IS NULL) AS hunt_unknown,
COUNT(*) FILTER (WHERE col_hunt_cert IS NULL) AS huntcert_unknown,
COUNT(*) FILTER (WHERE col_kill_flag IS NULL) AS kill_unknown
FROM clean.colobus;
</pre>

=== Solution ===

Approved for this conversion: make all three production columns nullable,
map <code>Y</code> to true and <code>N</code> to false, and preserve source NULL as SQL NULL. This
excludes no rows.

== (#204) COLOBUS FocalHuntCert can be unknown ==

=== Problem ===

There are 744 NULL focal-hunt-certainty values. Documentation described an
unknown state, but the production column was <code>NOT NULL</code>.

=== Bad Data ===

<pre>
SELECT COUNT(*)
FROM clean.colobus
WHERE col_focal_hunt_cert IS NULL;
</pre>

=== Solution ===

Approved for this conversion: resolve the schema/documentation conflict in
favor of the documented unknown state. Make <code>FocalHuntCert</code> nullable, map
<code>Y</code> and <code>N</code> to booleans, and preserve all source NULLs. This excludes no
rows.

== (#205) COLOBUS NumKills is absent for several distinct states ==

=== Problem ===

There are 389 rows without a total kill count: seven have <code>Kill = 'Y'</code>, 163
have <code>Kill = 'N'</code>, and 219 also have an unknown Kill flag. Seven meaningful
recorded male totals exceed the recorded overall total and must not be
rewritten.

=== Bad Data ===

<pre>
SELECT col_kill_flag, COUNT(*)
FROM clean.colobus
WHERE col_num_kills_total IS NULL
GROUP BY col_kill_flag
ORDER BY col_kill_flag NULLS LAST;
</pre>

=== Solution ===

Approved for this conversion: make <code>NumKills</code> nullable. Preserve every
recorded total; for a missing total map <code>Kill = 'Y'</code> to 1, <code>Kill = 'N'</code> to 0,
and unknown Kill to SQL NULL. Preserve the seven male/total differences and
verify source-to-target parity. This excludes no rows.

== (#206) COLOBUS FemaleNumKills has no supported derivation ==

=== Problem ===

The source does not provide an independently supported total female kill
count. Deriving it from other totals would invent data and can be invalid for
the seven rows where male kills exceed the overall total.

=== Bad Data ===

<pre>
SELECT COUNT(*)
FROM clean.colobus
WHERE col_num_kills_total IS NOT NULL
AND col_num_kills_by_males > col_num_kills_total;
</pre>

The query returns seven rows.

=== Solution ===

Approved for this conversion: make <code>FemaleNumKills</code> nullable and always load
SQL NULL. Do not derive it by subtraction. This excludes no rows.

== (#207) COLOBUS focal kill counts contain fractions ==

=== Problem ===

Three focal-male and three focal-female source counts have fractional values.
The production <code>INTEGER</code> columns cannot preserve them.

=== Bad Data ===

<pre>
SELECT COUNT(*) FILTER (
WHERE col_num_kills_by_focal_male <>
TRUNC(col_num_kills_by_focal_male)) AS focal_male,
COUNT(*) FILTER (
WHERE col_num_kills_by_focal_female <>
TRUNC(col_num_kills_by_focal_female)) AS focal_female
FROM clean.colobus;
</pre>

=== Solution ===

Approved for this conversion: change <code>FocalMaleNumKills</code> and
<code>FocalFemaleNumKills</code> to exact <code>NUMERIC</code> columns and preserve each source
value. This excludes no rows.

== (#208) COLOBUS group-size codes are unsupported and can be absent ==

=== Problem ===

The support table was empty. Source values use codes <code>1</code> through <code>5</code>, and 611
rows have no group-size code. Missing values must not be changed to code <code>4</code>,
which has a distinct recorded meaning.

=== Bad Data ===

<pre>
SELECT col_colobus_group_size, COUNT(*)
FROM clean.colobus
GROUP BY col_colobus_group_size
ORDER BY col_colobus_group_size NULLS LAST;
</pre>

The counts are 18, 3, 11, 1,555, and 49 for codes <code>1</code> through <code>5</code>, plus 611
NULLs.

=== Solution ===

Approved for this conversion: load the five documented support rows, retain
the foreign key, make <code>GroupSize</code> nullable, and preserve source NULL as SQL
NULL. The descriptions are <code>group consisted of only 1 col</code>, <code>group consisted
of only 2 col</code>, <code>group defined as small by observer</code>, <code>group but no size
estimate given</code>, and <code>group defined as large</code>. This excludes no rows.

== (#209) COLOBUS focal identifiers require approved corrections ==

=== Problem ===

Five source rows use four noncanonical focal identifiers. The approved
corrections are <code>GRP</code> to <code>NONE</code>, <code>STR</code> to <code>MGF</code>, <code>U</code> to <code>UNK</code>, and <code>FLT</code> to
<code>FLI</code>. <code>NONE</code> does not fit the source column's <code>VARCHAR(3)</code> declaration.

=== Bad Data ===

<pre>
SELECT col_fol_b_focal_chimp_id, COUNT(*)
FROM clean.colobus
WHERE col_fol_b_focal_chimp_id IN ('GRP', 'STR', 'U', 'FLT')
GROUP BY col_fol_b_focal_chimp_id
ORDER BY col_fol_b_focal_chimp_id;
</pre>

=== Solution ===

Approved for this conversion: widen only the clean copy of the focal column to
<code>TEXT</code>, then apply the four corrections to the five affected rows. Preserve
raw, tidy, and easy source fidelity. This correction itself excludes no
rows; the two corrected <code>UNK</code> rows are handled independently by Problem #210.

== * (#210) Corrected COLOBUS unknown focals have no valid watch context ==

=== Problem ===

After Problem #209 and all preceding watch loaders, the corrected <code>UNK</code> rows
at 1982-10-30 09:10 and 1985-07-21 10:12 have neither a B/Other watch nor a
covering <code>COMM_MEMBS</code> row. Inferring a community from unrelated same-day
observations would create unsupported data.

=== Bad Data ===

<pre>
SELECT col_encounter_date, col_fol_b_focal_chimp_id, col_begin_time
FROM clean.colobus
WHERE col_fol_b_focal_chimp_id = 'UNK'
AND (col_encounter_date, col_begin_time) IN (
(DATE '1982-10-30', TIME '09:10'),
(DATE '1985-07-21', TIME '10:12'))
ORDER BY col_encounter_date, col_begin_time;
</pre>

=== Solution ===

The investigator approved excluding exactly these two rows. Do not create a
watch with an inferred community. They do not overlap Problem #201, raising
the exact exclusion union from 15 to 17 rows. Of 2,247 source rows, 2,230 are
therefore eligible for conversion.

== * (#211) Corrected COLOBUS no-focal row has conflicting community context ==

=== Problem ===

After Problem #209 and all preceding watch loaders, the corrected <code>GRP</code> to
<code>NONE</code> row at 1978-09-10 11:33 has neither a B/Other watch nor a covering
<code>COMM_MEMBS</code> row. Its source comment says <code>ON PATROL, KAHAMA. NO FOCAL</code>, but
the unique same-day attendance community is <code>KK</code> (Kasekela); <code>HK</code> is Kahama
and its support notes say that community ended on 1977-12-31. Either community
choice would therefore infer unsupported data.

=== Bad Data ===

<pre>
SELECT col_encounter_date, col_fol_b_focal_chimp_id,
col_begin_time, col_comments
FROM clean.colobus
WHERE col_fol_b_focal_chimp_id = 'NONE'
AND col_encounter_date = DATE '1978-09-10'
AND col_begin_time = TIME '11:33';
</pre>

=== Solution ===

The investigator approved excluding exactly this row. Do not create a
no-focal Other watch with an inferred community. It does not overlap Problems
#201 or #210, raising the exact exclusion union from 17 to 18 rows. Of 2,247
source rows, 2,229 are therefore eligible for conversion.

== (#212) OTHER_SPECIES requires production species support codes ==

=== Problem ===

<code>codes.species</code> is empty, while <code>clean.other_species_lookup</code> contains 15
unique local-species names and unique English descriptions. Every
<code>SPECIES_PRESENT.Species</code> value must reference a nonempty uppercase
<code>codes.species.Species</code> value.

=== Bad Data ===

<pre>
SELECT UPPER(BTRIM(osl_local_species_name)) AS proposed_species,
osl_english_species_name AS proposed_description
FROM clean.other_species_lookup
ORDER BY proposed_species;
</pre>

The query returns 15 rows: <code>CHATU</code>, <code>FUNGO</code>, <code>KAKAKUONA</code>, <code>KENGE</code>, <code>KIMA</code>,
<code>MBOGO</code>, <code>NGURUWE</code>, <code>NKUNGE</code>, <code>NYANI</code>, <code>NYOKA</code>, <code>PONGO</code>, <code>TUMBILI</code>,
<code>UNKNOWN</code>, <code>UNRECORDED</code>, and <code>VYONDI</code>.

=== Solution ===

Approved by the investigator on 2026-09-18: populate <code>codes.species</code> using
the uppercase, trimmed local lookup name as <code>Species</code> and the corresponding
English lookup name as <code>Description</code>. Preserve all 15 descriptions exactly,
including <code>Mongoose? Civet? TBD</code>. Sanity must verify the exact 15 pairs and
their uniqueness before loading. This excludes no rows.

== (#213) OTHER_SPECIES species labels require canonical standardization ==

=== Problem ===

The source stores 30 distinct spellings of its classified local-species field.
Trimmed, case-insensitive matching resolves 6,360 of 6,366 rows to the 15-row
lookup. One additional row uses <code>Viondi</code>, a spelling variant of lookup value
<code>Vyondi</code>. The production code must use the uppercase canonical lookup value.

=== Bad Data ===

<pre>
SELECT quote_literal(os_osl_local_species_name) AS source_value,
COUNT(*) AS rows
FROM clean.other_species AS other_species
WHERE NOT EXISTS (
SELECT 1
FROM clean.other_species_lookup AS lookup
WHERE UPPER(BTRIM(lookup.osl_local_species_name)) =
UPPER(BTRIM(other_species.os_osl_local_species_name)))
GROUP BY os_osl_local_species_name
ORDER BY source_value;
</pre>

The query returns <code>Sokwe mgeni</code> (4), <code>UMBA </code> (1), and <code>Viondi</code> (1).

=== Solution ===

Approved by the investigator on 2026-09-18: standardize classified source
labels by trimmed, case-insensitive lookup and use the lookup's uppercase
local name as the production code. Correct the single <code>Viondi</code> value to
canonical <code>VYONDI</code>. Do not use <code>OS_local_species_name_written</code> to override
the classified field. This resolves 6,361 rows and excludes none. The five
remaining unlisted rows are handled independently by Problem #214.

== * (#214) OTHER_SPECIES contains unlisted species labels ==

=== Problem ===

Four source rows are classified as <code>Sokwe mgeni</code>, and one is classified as
<code>UMBA </code> with comment <code>'umba' written on sheet</code>. Neither trimmed label exists
in <code>clean.other_species_lookup</code> or has an approved <code>codes.species</code> code or
description.

=== Bad Data ===

<pre>
SELECT os_fol_date, os_fol_b_focal_animid, os_time_begin,
os_time_end, os_osl_local_species_name,
os_local_species_name_written, os_comments
FROM clean.other_species AS other_species
WHERE UPPER(BTRIM(os_osl_local_species_name)) IN ('SOKWE MGENI', 'UMBA')
ORDER BY os_fol_date, os_fol_b_focal_animid, os_time_begin;
</pre>

The five rows overlap neither the Problem #215 focal exclusion nor the Problem
#216 time exclusion.

=== Solution ===

Approved by the investigator on 2026-09-18: exclude exactly these five rows at
the loader boundary. Do not map either label to <code>UNKNOWN</code> or <code>UNRECORDED</code> and
do not create unsupported species codes. These rows overlap none of Problems
#215, #216, or #219 and contribute five rows to the final 13-row exclusion
union.

== * (#215) OTHER_SPECIES focal identifiers require normalization and one exclusion ==

=== Problem ===

Twelve rows have trailing spaces in <code>OS_FOL_B_focal_AnimID</code>, one row uses
lowercase <code>pl</code>, and one row uses <code>IMB</code>. The trimmed values resolve uniquely
to canonical production IDs for the 12 edge-spaced rows, and <code>pl</code> resolves
uniquely to <code>PL</code>. <code>IMB</code> is absent from <code>BIOGRAPHY_DATA</code> and has no watch.
Although an <code>IMA</code> B watch exists on the same date, that does not prove the two
identifiers name the same individual.

=== Bad Data ===

<pre>
SELECT DISTINCT quote_literal(os_fol_b_focal_animid) AS source_focal,
array_agg(DISTINCT biography_data.animid
ORDER BY biography_data.animid)
FILTER (WHERE biography_data.animid IS NOT NULL) AS canonical_matches
FROM clean.other_species
LEFT JOIN sokwedb.biography_data
ON LOWER(biography_data.animid) =
LOWER(BTRIM(other_species.os_fol_b_focal_animid))
WHERE os_fol_b_focal_animid <> BTRIM(os_fol_b_focal_animid)
OR NOT EXISTS (
SELECT 1
FROM sokwedb.biography_data AS exact_biography
WHERE exact_biography.animid =
other_species.os_fol_b_focal_animid)
GROUP BY os_fol_b_focal_animid
ORDER BY source_focal;
</pre>

=== Solution ===

Approved by the investigator on 2026-09-18: resolve trimmed values to the
single case-insensitive canonical <code>BIOGRAPHY_DATA.AnimID</code>, preserving its
stored spelling. This removes the 12 trailing spaces and maps <code>pl</code> to <code>PL</code>.
Exclude exactly the <code>IMB</code> row dated 2016-08-27 at 10:30; do not infer <code>IMA</code>.
Normalized source keys remain 6,366 unique. Exclude exactly the <code>IMB</code> row;
the independently missing B watch for <code>FU</code> is handled by Problem #219.

== * (#216) OTHER_SPECIES event times cannot satisfy production constraints ==

=== Problem ===

Six source rows cannot be represented without inventing or rewriting event
times. Three have NULL end times, two have Start after Stop, and one
additional row starts before 04:00. One reversed row also starts after 20:00.
There are no second-bearing times and no Stop values after 20:00.

=== Bad Data ===

<pre>
SELECT os_fol_date, os_fol_b_focal_animid,
os_time_begin, os_time_end, os_duration,
os_osl_local_species_name
FROM clean.other_species
WHERE os_time_end IS NULL
OR os_time_begin > os_time_end
OR os_time_begin < TIME '04:00'
OR os_time_begin > TIME '20:00'
OR os_time_end > TIME '20:00'
ORDER BY os_fol_date, os_fol_b_focal_animid, os_time_begin;
</pre>

=== Solution ===

Approved by the investigator on 2026-09-18: exclude the exact six-row union
at the loader boundary. Do not derive a missing endpoint from Duration or use
Duration to replace a recorded time. Preserve both endpoints for every other
row; <code>OS</code> events are intervals and cannot use <code>sdb_no_time</code>. These six rows
do not overlap Problems #214, #215, or #219. Together with Problems #214 and
#215 they bring the exclusion union to 12 rows.

== (#217) OTHER_SPECIES comments map to event notes ==

=== Problem ===

<code>EVENTS.Notes</code> is NOT NULL. Of 6,366 source rows, 6,352 have NULL
<code>OS_comments</code> and 14 have non-NULL comments. The separate
<code>OS_local_species_name_written</code> field contains free-form audit evidence but
has no production destination and sometimes differs from the classified
local-species field.

=== Bad Data ===

<pre>
SELECT COUNT(*) FILTER (WHERE os_comments IS NULL) AS null_comments,
COUNT(*) FILTER (WHERE os_comments IS NOT NULL) AS recorded_comments,
COUNT(*) FILTER (
WHERE os_comments IS NOT NULL
AND os_comments <> ''
AND BTRIM(os_comments) = '') AS whitespace_only_comments
FROM clean.other_species;
</pre>

The query returns 6,352 NULL comments, 14 recorded comments, and zero
nonempty whitespace-only comments.

=== Solution ===

Approved by the investigator on 2026-09-18: map <code>OS_comments</code> to
<code>EVENTS.Notes</code>, preserving non-NULL text exactly and mapping SQL NULL to <code>''</code>.
Retain <code>OS_local_species_name_written</code> in <code>clean</code> as audit evidence; do not
append it to event notes or use it to override the classified species field.
This excludes no rows. Validation must prove two-way parity under
<code>COALESCE(os_comments, '')</code>.

== (#218) SPECIES_PRESENT trigger error paths reference nonexistent fields ==

=== Problem ===

Both error-detail paths in
<code>db/schemas/lib/triggers/create/species_present.m4</code> reference
<code>NEW.researchers</code> and <code>NEW.nonresearchers</code>. Those are <code>HUMANS</code> columns;
<code>SPECIES_PRESENT</code> has only <code>EID</code> and <code>Species</code>. An invalid related event or a
conflicting <code>HUMANS</code> row can therefore obscure the intended integrity error
with a record-field error.

=== Bad Data ===

<pre>
grep -nE 'NEW\.(researchers|nonresearchers)' \
db/schemas/lib/triggers/create/species_present.m4
</pre>

The command reports four references, two in each error-detail path.

=== Solution ===

Resolved on 2026-09-18. Both invalid attempted-row details now report
<code>NEW.species</code>; the related event/watch and conflicting HUMANS details remain.
The generated trigger was installed in a disposable PostgreSQL 18.6 database,
and <code>db/tests/species_present_trigger.sql</code> passed rollback-only tests for wrong
event behavior, HUMANS conflict, a valid <code>OS</code> insert, immediate deferred
constraints, and fixture cleanup. This is a trigger repair and excludes no
source rows.

== * (#219) OTHER_SPECIES row has no B watch at the required load stage ==

=== Problem ===

After the normal conversion sequence through <code>load_follow_to_watches</code>, the
otherwise valid <code>FU</code> row dated 2015-12-13 at 16:45 has no B watch. <code>FU</code> has no
source FOLLOW row that day. Later aggression and grooming loaders create a B
watch for the same focal/date, but moving OTHER_SPECIES later would make its
prerequisite depend on unrelated observations and conceal the stage-specific
source gap.

=== Bad Data ===

<pre>
SELECT os_fol_date, os_fol_b_focal_animid, os_time_begin, os_time_end,
os_osl_local_species_name, os_comments
FROM clean.other_species
WHERE os_fol_date = DATE '2015-12-13'
AND os_fol_b_focal_animid = 'FU'
AND os_time_begin = TIME '16:45';

SELECT watches.wid
FROM sokwedb.watches
WHERE watches.animid = 'FU'
AND watches.date = DATE '2015-12-13'
AND watches.type = 'B';
</pre>

At the required post-follow loader stage, the first query returns the one
<code>VYONDI</code> interval from 16:45 through 17:00 and the second query returns no row.

=== Solution ===

Approved by the investigator on 2026-09-18: exclude exactly the <code>FU</code> row dated
2015-12-13 at 16:45. Keep OTHER_SPECIES immediately after
<code>load_follow_to_watches</code>; do not create a watch and do not reuse a watch that
only appears after later aggression or grooming loads. This row overlaps
none of Problems #214, #215, or #216, increasing the final exclusion union to
13 rows and leaving 6,353 rows eligible for conversion.

== (#220) FOLLOW_MAP_LOCATION rows can contain two location representations ==

=== Problem ===

Of 1,823,766 <code>FOLLOW_MAP_LOCATION</code> rows, 703,807 contain both a paper map
sequence and paired UTM coordinates. Production stores PAPER and UTM details
on separate EVENTS rows, so treating the source row as a choice of one detail
table would discard one recorded representation.

=== Bad Data ===

The read-only <code>conversion/follow_map_location_profile.sql</code> reports 703,807
dual rows, 69 paper-only rows, 1,119,890 UTM-only rows, and no row with neither
representation. Before exclusions this is 703,876 PAPER representations and
1,823,697 UTM representations.

=== Solution ===

Approved by the investigator on 2026-09-18: emit one PAPER event and one UTM
event for every eligible dual row. Both events use the same location watch
and source time. No loader may choose one representation or infer that UTM
coordinates accompanying a paper map are direct GPS observations.

== * (#221) FOLLOW_MAP_LOCATION focal identities are missing or unsupported ==

=== Problem ===

Location watches require a real <code>BIOGRAPHY_DATA.AnimID</code>; <code>NONE</code> is prohibited.
The refreshed source has 6,509 NULL-focal rows and 7,495 rows in 33 trimmed
labels having no normalized biography match. The source does not contain
evidence from which to invent their identities.

Fourteen noncanonical spellings covering 609 rows do resolve uniquely under
<code>lower(normalize(btrim(value)))</code>. They include edge-space variants and the
case variants <code>Nas</code> to <code>NAS</code>, <code>fd</code> to <code>FD</code>, <code>pl</code> to <code>PL</code>, and <code>sw </code> to <code>SW</code>.

=== Bad Data ===

Run <code>conversion/follow_map_location_profile.sql</code>. It prints every unmatched
trimmed label with its row count and date range, and reports 214 raw non-NULL
spellings, 204 trimmed spellings, and 200 normalized spelling classes. The
unmatched labels include punctuation-prefixed labels, apparent suffixed IDs,
and labels that cannot safely be corrected from spelling alone.

=== Solution ===

Approved by the investigator on 2026-09-18: standardize the 609 uniquely
matched rows to the stored biography spelling and exclude all 6,509 NULL-focal
rows and 7,495 unmatched-label rows. This excludes 14,004 rows before overlap
with other problems. Do not derive a focal from coordinates, community, date,
another observation, or a nearby follow.

== * (#222) FOLLOW_MAP_LOCATION event times cannot be represented ==

=== Problem ===

PAPER and UTM events require a recorded minute from 04:00 through 20:00. The
refreshed source has 25,921 NULL times, 676 times before 04:00, and 258 after
20:00. One recorded time is midnight, so midnight cannot represent NULL.

=== Bad Data ===

<code>conversion/follow_map_location_profile.sql</code> reports the exact classes and
representation impact. NULL times affect 25,921 PAPER and 25,914 UTM
representations; early times affect 676 UTM representations; late times
affect 7 PAPER and 258 UTM representations. Recorded seconds are always zero.

=== Solution ===

Approved by the investigator on 2026-09-18: exclude all 25,921 NULL-time rows,
676 rows before 04:00, and 258 rows after 20:00. This excludes 26,855 rows
before overlap with other problems. Do not map NULL to midnight, clamp
out-of-range times, round times, or derive a time from neighboring source rows.

== * (#223) FOLLOW_MAP_LOCATION event keys collide ==

=== Problem ===

Production allows at most one PAPER and one UTM event for a location watch at
a given time. Before focal normalization and after omitting NULL times, the
source has 23,801 colliding PAPER focal/date/time keys and 25,561 colliding UTM
keys. Generated IDs cannot make those keys representable.

=== Bad Data ===

The profile reports 47,603 PAPER rows in colliding keys, with 23,802 excess
rows, and 51,255 UTM rows, with 25,694 excess rows. Only 75 PAPER keys (150
rows) and 150 UTM keys (300 rows) have one identical representation value;
23,726 PAPER keys and 25,411 UTM keys contain genuinely different recorded
representations. One UTM key has 14 source rows.

The complete source additionally has 151 exact duplicate pairs across all 14
columns. A generated conversion ID does not establish that either copy is a
separate production observation.

=== Solution ===

Approved by the investigator on 2026-09-18: exclude every source row whose
canonical focal/date/time key collides for either available representation.
This excludes the entire source row, rather than retaining half of a dual row,
and removes 51,457 rows before overlap with other problems. Do not add
arbitrary time offsets, select a row by generated order, or let <code>ON CONFLICT</code>
discard rows.

== * (#224) FOLLOW_MAP_LOCATION community recovery counts depend on focal normalization ==

=== Problem ===

The investigator approved dated <code>COMM_MEMBS</code> as canonical, <code>Unknown</code> only for
a NULL source community without dated membership, and exclusion of unsupported
non-NULL communities and valid non-NULL communities without matching dated
membership or disagreeing with membership.

The previously approved exact counts used a case-sensitive
<code>btrim(focal) = COMM_MEMBS.AnimID</code> join. The loader specification instead
requires canonical focal matching by <code>lower(normalize(btrim(value)))</code>.
Applying that rule resolves 118 additional rows through <code>Nas</code> to <code>NAS</code>, <code>fd</code>
to <code>FD</code>, <code>pl</code> to <code>PL</code>, and <code>sw </code> to <code>SW</code>.

=== Bad Data ===

On the refreshed source, canonical matching reports:

* 1,095,525 NULL-source rows with membership;
* 14,412 NULL-source rows without membership;
* 712,457 valid-source rows agreeing with membership;
* 245 valid-source rows without membership;
* 1,105 valid-source rows disagreeing with membership; and
* 22 unsupported non-NULL source communities.

The canonical result would exclude 1,372 rows rather than 1,455. The old
trim-only result remains exactly 1,095,490, 14,447, 712,374, 328, 1,105, and
22 respectively. The 83-row exclusion difference is 51 <code>fd</code>, 14 <code>pl</code>, and 18
<code>sw </code> rows whose valid source community agrees with canonical membership. The
other 35 newly resolved rows are NULL-community <code>Nas</code> rows and would change
from <code>Unknown</code> to membership community <code>KK</code>.

=== Solution ===

Canonical focal standardization was approved under Problem #221. The final
Problem #224 exclusion union is therefore 1,372 rows: 22 unsupported non-NULL
communities, 245 valid communities without dated membership, and 1,105 valid
communities disagreeing with membership. Do not preserve the old counts by
using a less canonical identity join.

== (#225) LOCATION_ORIGINS lacks approved support values ==

=== Problem ===

<code>LOCATION_ORIGINS</code> is empty, but every PAPER and UTM detail requires an origin.
The source contains <code>GPS</code> on 1,113,996 rows, <code>Paper Map</code> on 15,165 rows,
<code>paper maps</code> on 27,273 rows, and NULL on 667,332 rows.

=== Bad Data ===

The profile reports that NULL origin affects 661,438 paper representations and
667,332 UTM representations. Of the latter, 5,894 are UTM-only rows. A UTM
representation therefore does not prove GPS origin, and dual historical rows
may contain coordinates converted from a paper map.

=== Solution ===

Approved by the investigator on 2026-09-18. Populate canonical origins <code>GPS</code>
(<code>GPS</code>), <code>PAPER</code> (<code>Paper map</code>), and <code>UNKNOWN</code> (<code>Unknown</code>). The uppercase code
is required by the production support-table constraint. Map source <code>GPS</code> to
<code>GPS</code>, both <code>Paper Map</code> and <code>paper maps</code> to <code>PAPER</code>, and SQL NULL to <code>UNKNOWN</code>.
Apply the same source mapping to both representations; do not infer GPS solely
from the presence of coordinates.

== (#226) FOLLOW_MAP_LOCATION entered-person support is incomplete ==

=== Problem ===

<code>Entered</code> is NULL on 1,781,328 rows. Recorded values include two labels for
Danielle Lodge and the full name <code>Julianna Turner</code>; those values did not
previously have complete PEOPLE support.

=== Bad Data ===

The refreshed profile reports 17,794 <code>DL</code>, 8,902 <code>Danielle Lodge</code>, 7,968
<code>Julianna Turner</code>, 2,314 <code>KS</code>, 4,806 <code>SM</code>, and 654 <code>SR</code> rows. The refreshed
support table contains <code>UNK</code>, <code>KS</code>, <code>SM</code>, <code>SR</code>, <code>DL</code>, and <code>Julianna Turner</code>
with no normalized duplicate.

=== Solution ===

Approved by the investigator on 2026-09-18. Map NULL to <code>UNK</code>; map both <code>DL</code>
and <code>Danielle Lodge</code> to <code>DL</code>; map <code>Julianna Turner</code> to that same-named code;
and preserve <code>KS</code>, <code>SM</code>, and <code>SR</code>. Match with
<code>lower(normalize(btrim(value)))</code>. The manually maintained <code>clean.people</code> seed
contains <code>DL</code> and <code>Julianna Turner</code>, both described as follow map location data
entry. This decision excludes no rows.

== (#227) FOLLOW_MAP_LOCATION contains spatial anomalies ==

=== Problem ===

Thirty-nine UTM rows have coordinates outside the expected local orientation,
and two rows have elevation 20,000. The production numeric limits equal the
source extrema, so constraint acceptance is not evidence of geographic
validity.

=== Bad Data ===

<code>conversion/follow_map_location_profile.sql</code> prints all 41 anomaly rows. The
39 coordinate rows include equal axes, axes both near 9.48 million, and values
that appear to have missing digits. The two elevation rows are dated
2020-03-08 (<code>KOM</code>) and 2021-06-06 (<code>AP</code>).

=== Solution ===

Approved by the investigator on 2026-09-18: preserve all 39 unusual coordinate
rows and both elevation-20,000 rows exactly as recorded. They satisfy current
production bounds. Do not swap axes, add digits, replace elevation by
heuristic, or exclude these rows solely because they are anomalous.

== (#228) FOLLOW_MAP_LOCATION has NULL optional metadata ==

=== Problem ===

Production <code>FollowNum</code> and <code>Notes</code> are NOT NULL but allow empty text. Source
<code>FollowNum</code> is NULL on 1,163,270 rows and <code>Notes</code> is NULL on 1,823,410 rows.
<code>fml_update</code> is populated on 48,796 rows but has no production destination.

=== Bad Data ===

The profile reports 18,541 distinct populated follow numbers of length 5
through 10, and 356 populated notes containing 197 distinct values. Every
populated value can be preserved exactly. Every populated <code>fml_update</code> is
2018-05-20.

=== Solution ===

Approved by the investigator on 2026-09-18: map SQL NULL <code>FollowNum</code> and
<code>Notes</code> to empty text while preserving populated values exactly. Retain
<code>fml_update</code> only in the clean audit source and do not append it to notes.

== * (#229) FOLLOW_MAP_LOCATION dates exceed focal departure dates ==

=== Problem ===

<code>WATCHES</code> rejects a watch before the focal's birth or entry and after the
focal's departure from study. The source has 229 canonically matched rows
after <code>BIOGRAPHY_DATA.DepartDate</code>, of which 47 are already excluded by both
Problems #222 and #224. The approved Problems #220-#228 projection therefore
still contained 182 rows after departure. There are no projected rows before
birth or entry.

All 182 affected rows are UTM-only GPS records with NULL source community,
NULL entered person, and valid event times. They form two location watches:

* focal <code>S</code>, 134 rows on 2017-11-29 from 06:46 through 18:42, after departure
date 1968-01-19; and
* focal <code>DE</code>, 48 rows on 2017-12-27 from 12:41 through 17:02, after departure
date 1974-05-07.

=== Bad Data ===

<pre>
SELECT source.*,
biography_data.birthdate,
biography_data.entrydate,
biography_data.departdate
FROM clean.follow_map_location AS source
JOIN sokwedb.biography_data
ON lower(normalize(biography_data.animid)) =
lower(normalize(btrim(source.fml_fol_b_focal_animid)))
WHERE source.fml_fol_date < biography_data.birthdate
OR source.fml_fol_date < biography_data.entrydate
OR source.fml_fol_date > biography_data.departdate
ORDER BY source.fml_conversion_id;
</pre>

The first full chunked load stopped when inserting the <code>S</code> location watch on
2017-11-29 because the <code>WATCHES</code> trigger reported its 1968-01-19 departure.

=== Solution ===

Approved by the investigator on 2026-09-18: exclude all post-departure rows.
The predicate identifies 229 raw rows; 47 overlap both Problems #222 and #224,
so Problem #229 adds exactly 182 rows to the exclusion union. The final
distinct union is 93,470, leaving 1,730,296 eligible source rows. Do not weaken
the production biography-date trigger, substitute another focal, or alter the
recorded location dates.

== (#230) Location events lack a matching focal arrival interval ==

=== Problem ===

Production documentation expects location times to fall within a focal's
arrival interval on a B watch, but the database reports this condition as a
warning rather than rejecting the event. Of 2,358,936 loaded PAPER and UTM
events, 779,682 do not fall within a matching focal arrival interval.

The refreshed post-load profile separates them as follows:

* 724,075 events have no B watch for the focal/date: 196 PAPER and 723,879 UTM;
* 404 events have a B watch but no ARR interval: 202 PAPER and 202 UTM; and
* 55,203 events are outside every ARR interval on an existing B watch: 19,118
PAPER and 36,085 UTM.

These classes cover 1973-01-04 through 2022-11-28. The large no-B-watch UTM
class reflects location coverage continuing well beyond the consolidated
follow data; it is not evidence from which to invent a follow or arrival time.

=== Bad Data ===

Join location <code>EVENTS</code> and their <code>L</code> <code>WATCHES</code> rows to B watches on focal/date,
then test whether the location start falls between an ARR event's start and
stop. <code>conversion/load_follow_map_location_validation.sql</code> reports the exact
779,682-event total after proving source-to-target parity.

=== Solution ===

Approved by the investigator on 2026-09-18: preserve all 779,682 events as
recorded. The production schema permits them and the arrival relationship is
documentation guidance enforced as a warning. Validation asserts the exact
count and continues to report it. Do not create B watches, manufacture ARR
intervals, alter location times, or exclude otherwise eligible source rows to
satisfy this guidance.

== * (#231) CHIMPS_ON_TIKIS source identifiers lack biography support ==

=== Problem ===

Three <code>clean.chimps_on_tikis_lookup</code> rows cannot satisfy the required
<code>CHIMPS_ON_TIKIS.AnimID</code> foreign key: <code>MGENI</code> occurs once in <code>KK</code> and once in
<code>MT</code>, and <code>VAN mtoto</code> occurs once in <code>KK</code>. <code>MGENI</code> does not identify a
particular production individual. <code>VAN mtoto</code> is not a production AnimID.

<code>VIS</code> is VAN's only supported child alive on the Tiki creation and deployment
dates and had <code>KK</code> membership, but that evidence does not establish that the
source label denotes <code>VIS</code>.

=== Bad Data ===

<pre>
SELECT source.*
FROM clean.chimps_on_tikis_lookup AS source
LEFT JOIN sokwedb.biography_data AS biography
ON biography.animid = source.b_animid
WHERE biography.animid IS NULL
ORDER BY source.tiki_creation_date,
source.community_id,
source.b_animid;
</pre>

The dated source returns exactly:

<pre>
b_animid | tiki_creation_date | community_id | tiki_deployment_date | new_addition
-----------+--------------------+--------------+----------------------+-------------
MGENI | 2020-08-04 | KK | 2020-08-20 | N
VAN mtoto | 2020-08-04 | KK | 2020-08-20 | Y
MGENI | 2020-08-04 | MT | 2020-08-20 | N
</pre>

=== Solution ===

Approved by the investigator on 2026-09-19: exclude exactly these three
complete source tuples. Do not create a biography row for <code>MGENI</code>, infer an
individual for either <code>MGENI</code> row, or map <code>VAN mtoto</code> to <code>VIS</code>. The loader
sanity check must prove the exact exclusion inventory and reject any remaining
unsupported identifier. The dated source has 84 rows, leaving 81 eligible
rows.

Implemented and verified on 2026-09-19. The shared projection compares all
five source columns with <code>IS NOT DISTINCT FROM</code> and excludes only the approved
tuples. Read-only sanity reproduced 84 source rows, the exact three-row
exclusion multiset, and 81 supported unique projected rows. The direct load
produced 81 target rows, including 13 <code>TRUE</code> and 68 <code>FALSE</code> newcomer values;
bidirectional <code>EXCEPT ALL</code> returned no payload differences. An intentional
post-insert failure rolled all 81 rows back to zero. An approved clean retry
reproduced the same payload digest, <code>28e17175eba7622cbf9afc7130dc1900</code>.

== (#232) CHIMPS_ON_TIKIS deployment-date index targets AnimID ==

=== Problem ===

The index named <code>chimps_on_tikis_tikideployementdate</code> is defined on <code>animid</code>
instead of <code>tikideploymentdate</code>. It duplicates the separate AnimID index and
leaves the documented deployment-date lookup unindexed. The index name also
misspells "deployment".

=== Bad Data ===

<pre>
SELECT indexname, indexdef
FROM pg_indexes
WHERE schemaname = 'housekeeping'
AND tablename = 'chimps_on_tikis'
AND indexname = 'chimps_on_tikis_tikideployementdate';
</pre>

The live PostgreSQL 18.6 database reports an index using <code>btree (animid)</code>.
The owning definition is in
<code>db/schemas/housekeeping/indexes/create/chimps_on_tikis.m4</code>; the corresponding
drop name is in the neighboring drop M4.

=== Solution ===

Approved by the investigator on 2026-09-19: before loading this table, replace
the defective index with correctly spelled
<code>chimps_on_tikis_tikideploymentdate</code> on <code>(tikideploymentdate)</code>. Update both
owning M4 files, regenerate owned index SQL artifacts, and verify the live
index definition in a disposable database. This schema prerequisite excludes
no source rows.

Implemented and verified on 2026-09-19. The owning create and drop M4 files
now use the corrected index, while the drop definition also removes the legacy
misspelled name during an in-place migration. Repository generation rebuilt
the per-object and aggregate index SQL. Both the focused local migration and
the freshly rebuilt disposable integration schema reported exactly one index
on <code>tikideploymentdate</code>, named
<code>chimps_on_tikis_tikideploymentdate</code>, and no legacy index.

== * (#233) SIV_STATUS_BOUT comma-bearing names shifted into later columns ==

=== Problem ===

Five source rows have a comma-bearing original name split across
<code>SSB_names_used</code>, <code>SSB_in_bio_table</code>, and <code>SSB_notes</code>. The leading name
fragment is in <code>SSB_names_used</code>, the trailing fragment is in
<code>SSB_in_bio_table</code>, and the actual biography flag <code>1</code> is in <code>SSB_notes</code>.
These malformed values already occur in <code>easy.siv_status_bout</code>; they were not
introduced by a clean-stage transformation.

=== Bad Data ===

<pre>
SELECT source.*
FROM clean.siv_status_bout AS source
WHERE source.ssb_in_bio_table NOT IN ('0', '1')
ORDER BY source.ssb_b_animid, source.ssb_start_date;
</pre>

The dated source returns exactly five rows: <code>DB</code> with fragments <code>"Darbee</code> and
<code>NA"</code>; <code>TOF</code> with <code>"Tofiki</code> and <code>gone"</code>; and all three <code>TT</code> bouts with
<code>"Kati (Tita)</code> and <code>Kati-tita"</code>. Each row has <code>SSB_notes = '1'</code>.

=== Solution ===

Approved by the investigator on 2026-09-19: exclude these five exact complete
source tuples. The final decision supersedes an earlier consideration of
reconstructing the values in <code>clean</code>. Do not repair the imported fields,
infer punctuation, or use a broad identifier-only exclusion. Sanity must
prove the exact five-row multiset, and the canonical projection must compare
all ten source columns with NULL-safe equality.

Implemented and verified on 2026-09-19. <code>conversion/siv_status_bout_projection.sql</code>
encodes the exact five-row <code>DB</code>/<code>TOF</code>/<code>TT</code> tuple set, including the two
literal embedded double-quote characters recovered with <code>quote_literal</code>.
Read-only sanity, run against the local PostgreSQL 18.6 <code>sokwedb</code> database,
reproduced this exact multiset with bidirectional <code>EXCEPT ALL</code> and confirmed
it overlaps Problem #234 in exactly one row and Problem #235 in none.

== * (#234) SIV_STATUS_BOUT required test dates are missing ==

=== Problem ===

Twelve source rows cannot satisfy the target's non-NULL <code>FirstTestDate</code> and
<code>LastTestDate</code> columns. All 11 <code>unk</code> bouts have both dates absent. The <code>VID</code>
positive bout has <code>FirstTestDate = 2000-09-09</code> and a NULL <code>LastTestDate</code>.

=== Bad Data ===

<pre>
SELECT source.*
FROM clean.siv_status_bout AS source
WHERE source.ssb_first_test_date IS NULL
OR source.ssb_last_test_date IS NULL
ORDER BY source.ssb_b_animid, source.ssb_start_date;
</pre>

The 11 unknown bouts belong to <code>BIM</code>, <code>CH100</code>, <code>ERI</code>, <code>FO</code>, <code>GIM</code>, <code>HAI</code>,
<code>MAK</code>, <code>RUD</code>, <code>TT</code>, <code>YD</code>, and <code>ZS</code>. The <code>TT</code> unknown bout is also one of the
five Problem #233 shifted rows.

=== Solution ===

Approved by the investigator on 2026-09-19: exclude the 12 exact complete
source tuples. Do not fill absent tests from bout boundaries and do not change
target nullability. Match all ten source columns with NULL-safe equality.
Because the <code>TT</code> row overlaps Problem #233, count it once in the combined
exclusion set.

This decision removes every source <code>unk</code> bout. Ten retained positive rows
therefore keep <code>BoutNumber = 3</code> without a loaded bout 2. Preserve those source
bout numbers; do not renumber or synthesize replacement bouts. Update the
production documentation to report the resulting snapshot accurately.

Implemented and verified on 2026-09-19. The projection excludes the exact
12-row tuple set, including the <code>TT</code> row shared with Problem #233. Sanity
reproduced 11 NULL <code>FirstTestDate</code> rows, 12 NULL <code>LastTestDate</code> rows, and
confirmed the resulting 164-row projection contains zero <code>unk</code> rows and
exactly 10 <code>BoutNumber = 3</code> rows with no corresponding loaded
<code>BoutNumber = 2</code> row, including <code>CH100</code>, whose eligible negative and
positive bouts are retained.

== * (#235) SIV_STATUS_BOUT AMA incorrectly claims biography support ==

=== Problem ===

The source row for <code>AMA</code> has <code>SSB_in_bio_table = '1'</code>, but no exact <code>AMA</code>
identifier exists in <code>BIOGRAPHY_DATA</code>. <code>AME</code> exists and the source note says
the row includes <code>AME</code> samples, but that is not sufficient evidence to replace
the source identifier.

=== Bad Data ===

<pre>
SELECT source.*
FROM clean.siv_status_bout AS source
WHERE source.ssb_b_animid = 'AMA'
AND source.ssb_in_bio_table = '1'
AND NOT EXISTS (
SELECT 1
FROM sokwedb.biography_data AS biography
WHERE biography.animid = source.ssb_b_animid);
</pre>

The dated source returns one <code>AMA</code> negative bout from 2014-03-20 through
2019-04-05, with tests from 2017-01-31 through 2019-04-05 and bout number 1.

=== Solution ===

Approved by the investigator on 2026-09-19: exclude this exact complete source
tuple. Do not map <code>AMA</code> to <code>AME</code>, recompute the flag, or create biography
support. Preserve all other unsupported source identifiers whose flags are
<code>0</code>, as the target intentionally permits identifiers outside biography.

Implemented and verified on 2026-09-19. The projection excludes the exact
<code>AMA</code> tuple, including its literal doubled-double-quote note text. Sanity
confirmed the 21 remaining unsupported identifiers (<code>CH064</code> through <code>CH128</code>
and <code>TUM</code>) are retained with <code>InBioTable = FALSE</code> and that no eligible row
has an <code>InBioTable</code>/biography-support mismatch.

== (#236) SIV_STATUS_BOUT absent notes cannot satisfy the target ==

=== Problem ===

The source contains 169 NULL <code>SSB_notes</code> values, while target <code>Notes</code> is
non-NULL. The target's <code>notonlyspaces_check</code> explicitly permits the empty
string as the representation of no note.

After the approved Problems #233 through #235 exclusion union, 158 eligible
rows have NULL notes and six have non-NULL notes.

=== Bad Data ===

<pre>
SELECT COUNT(*)
FROM clean.siv_status_bout AS source
WHERE source.ssb_notes IS NULL;
</pre>

The dated source returns 169.

=== Solution ===

Approved by the investigator on 2026-09-19: map an eligible NULL note to the
empty string with <code>COALESCE(source.ssb_notes, '')</code>. Preserve every non-NULL
note exactly. Sanity and validation must prove that this is the only notes
transformation and that no projected note contains only spaces.

Problems #233, #234, and #235 contain 5, 12, and 1 rows respectively. One
<code>TT</code> row overlaps #233 and #234, so their union excludes 17 distinct rows from
181 and leaves 164 rows for conversion.

Implemented and verified on 2026-09-19. <code>COALESCE(source.ssb_notes, '')</code> is
the projection's only notes transformation. Read-only sanity and validation
against the local PostgreSQL 18.6 <code>sokwedb</code> database confirmed 158 loaded
empty notes came only from NULL source notes, all 6 non-NULL notes are
preserved byte-for-byte, and no loaded note contains only spaces.

The direct set-based loader
(<code>conversion/load_siv_status_bout.sql</code>) inserted exactly 164 rows: 132 <code>neg</code>,
32 <code>pos</code>, 0 <code>unk</code>; 154 <code>BoutNumber = 1</code>, 0 <code>BoutNumber = 2</code>, 10
<code>BoutNumber = 3</code>; 142 <code>TRUE</code> and 22 <code>FALSE</code> <code>InBioTable</code> values; and 158
empty/6 nonempty notes. Bidirectional <code>EXCEPT ALL</code> between the canonical
projection and the loaded rows returned no differences in either direction.
An intentional post-insert failure (<code>SELECT 1/0</code>) rolled the single
transaction back to zero target rows. An authorized <code>TRUNCATE</code> of only the
disposable target table followed by an unchanged sanity/load/validation
sequence reproduced the identical sorted ten-column payload digest,
<code>8b243c9b7f53d79f9fa1eae640e26b51</code>, on both runs. <code>git diff --check</code>, a
local-only Make dry run of the full <code>load_data</code> sequence, and a direct <code>m4</code>
and repository doc-build render of <code>doc/src/housekeeping/siv_status_bout.m4</code>
all passed without error.

== (#237) SUBADULT_ARRIVALS_LOG FirstTikiDate is declared NOT NULL contrary to its own documentation ==

=== Problem ===

<code>db/schemas/housekeeping/tables/create/subadult_arrivals_log.m4</code> declares
<code>firsttikidate DATE NOT NULL</code>, but the owning documentation, added in the
same commit, says the column "may be NULL when there is no information on
the date of permanent addition." 156 of 209 source rows (75%) have a NULL
<code>SA_first_tiki_date</code>, each carrying an explanatory note describing why no
discrete tiki date applies. A NULL <code>FirstTikiDate</code> is this table's normal,
permanent representation for most individuals, not missing data awaiting
entry.

=== Bad Data ===

<pre>
SELECT count(*) FILTER (WHERE sa_first_tiki_date IS NULL) AS null_date
, count(*) FILTER (WHERE sa_first_tiki_date IS NOT NULL) AS dated
FROM clean.subadult_arrivals_log;
</pre>

The dated source returns 156 NULL and 53 non-NULL rows. <code>git show cc3640b
--stat</code> shows the table, its documentation, its indexes, and its trigger
were all added in one commit, so this contradiction has existed since the
table was created and was never exercised by a loader.

=== Solution ===

Approved by the investigator on 2026-09-19: correct the schema defect
rather than exclude the 156 legitimately NULL-dated rows. Remove <code>NOT
NULL</code> from the <code>firsttikidate</code> column in the owning create M4, regenerate
the owned table SQL artifact, and verify the corrected nullability in a
disposable or migrated database before loading. Leave the unique <code>AnimID</code>
index, the <code>FirstTikiDate</code> index, the <code>AnimID</code> foreign key and <code>NONE</code>
check, and the update trigger unchanged. This is a target-schema
correction, not a clean-stage mutation, and excludes no source rows on its
own.

Implemented and verified on 2026-09-19. <code>NOT NULL</code> was removed from
<code>firsttikidate</code> in
<code>db/schemas/housekeeping/tables/create/subadult_arrivals_log.m4</code>; the owned
table SQL artifact was regenerated with <code>make db/schemas/createtables.sql</code>
and shows <code>firsttikidate DATE</code> with no other column or constraint changed.
<code>ALTER TABLE housekeeping.subadult_arrivals_log ALTER COLUMN firsttikidate
DROP NOT NULL</code> was applied to the local PostgreSQL 18.6 <code>sokwedb</code> disposable
database, and <code>information_schema.columns</code> confirmed <code>firsttikidate</code> is
nullable while <code>id</code>, <code>animid</code>, and <code>notes</code> remain <code>NOT NULL</code> and the unique
<code>AnimID</code> index, <code>FirstTikiDate</code> index, <code>AnimID</code> foreign key and <code>NONE</code>
check, and update trigger are unchanged.

== * (#238) SUBADULT_ARRIVALS_LOG CT's FirstTikiDate follows CT's own departure date ==

=== Problem ===

<code>CT</code>'s parsed <code>FirstTikiDate</code> (1986-03-01) is after <code>CT</code>'s own
<code>BIOGRAPHY_DATA.DepartDate</code> (1985-09-30). The target has no constraint
tying <code>FirstTikiDate</code> to biography dates, so this row would load without
error, but a tiki date recorded after an individual's departure is not
trustworthy evidence of when that individual was added to the tiki sheets.

=== Bad Data ===

<pre>
SELECT source.*, biography.departdate
FROM clean.subadult_arrivals_log AS source
JOIN sokwedb.biography_data AS biography
ON biography.animid = source.sa_b_animid
WHERE to_date(source.sa_first_tiki_date, 'MM/DD/YY') > biography.departdate;
</pre>

The dated source returns exactly one row: <code>CT</code>, <code>03/01/86</code>, with note "Btwn
DOB and CA death, arrivals same as CA; Elo confirmed that always accounted
for on tikis afterwards." No other row falls outside its individual's
birth/entry/departure window.

=== Solution ===

Approved by the investigator on 2026-09-19: exclude this exact row rather
than preserve it as a documented semantic anomaly. Match on the exact
source tuple (<code>sa_b_animid</code>, <code>sa_first_tiki_date</code>, <code>sa_notes</code>) with
NULL-safe equality; do not exclude by <code>AnimID</code> alone or use a broad
date-comparison predicate as the operative exclusion, and do not infer a
corrected date. This is the only approved exclusion; it leaves 208
eligible rows.

Implemented and verified on 2026-09-19.
<code>conversion/subadult_arrivals_log_projection.sql</code> encodes the exact
one-row <code>CT</code> tuple. Read-only sanity, run against the local PostgreSQL
18.6 <code>sokwedb</code> database, reproduced this exact tuple with bidirectional
<code>EXCEPT ALL</code> and confirmed no other eligible row falls outside its
individual's birth/entry/departure window.

== (#239) SUBADULT_ARRIVALS_LOG absent notes cannot satisfy the target ==

=== Problem ===

The source contains 6 NULL <code>SA_notes</code> values, all outside the Problem #238
exclusion. The target <code>Notes</code> is NOT NULL but explicitly permits the empty
string.

=== Bad Data ===

<pre>
SELECT count(*)
FROM clean.subadult_arrivals_log
WHERE sa_notes IS NULL;
</pre>

The dated source returns 6.

=== Solution ===

Approved by the investigator on 2026-09-19: map an eligible NULL note to
the empty string with <code>COALESCE(source.sa_notes, '')</code>. Preserve every
non-NULL note exactly, including two rows (<code>FAD</code>, <code>KEA</code>) that carry leading
or trailing spaces around substantive text. Sanity and validation must
prove that this is the only notes transformation and that no projected
note contains only spaces. This mapping, together with Problem #237's
schema fix and Problem #238's exclusion, leaves 208 rows for conversion:
209 source rows minus the one Problem #238 exclusion.

Implemented and verified on 2026-09-19.
<code>COALESCE(source.sa_notes, '')</code> is the projection's only notes
transformation. Read-only sanity and validation against the local
PostgreSQL 18.6 <code>sokwedb</code> database confirmed 6 loaded empty notes came
only from NULL source notes, all 202 non-NULL notes are preserved
byte-for-byte (including the two rows with leading/trailing spaces), and
no loaded note contains only spaces.

The direct set-based loader (<code>conversion/load_subadult_arrivals_log.sql</code>)
inserted exactly 208 rows: 156 NULL and 52 non-NULL <code>FirstTikiDate</code>
values (extent 1976-05-01 through 2013-09-22, unchanged by the <code>CT</code>
exclusion), and 6 empty/202 nonempty notes. Every non-NULL <code>FirstTikiDate</code>
was proven to round-trip through <code>to_char(..., 'MM/DD/YY')</code> back to its
exact source text. Bidirectional <code>EXCEPT ALL</code> between the canonical
projection and the loaded rows returned no differences in either
direction. An intentional post-insert failure (<code>SELECT 1/0</code>) rolled the
single transaction back to zero target rows. An authorized <code>TRUNCATE</code> of
only the disposable target table followed by an unchanged
sanity/load/validation sequence reproduced the identical sorted
four-column payload digest, <code>6eac68b10ebb802e3c78b9109e28d3d7</code>, on both
runs. <code>git diff --check</code>, a local-only Make dry run of the full
<code>load_data</code> sequence, and a direct <code>m4</code> and repository doc-build render
of <code>doc/src/housekeeping/subadult_arrivals_log.m4</code> all passed without
error.

== (#240) TLK_BRECORD_NOTES_CODES two Abbreviation values carry a trailing space ==

=== Problem ===

<code>"ID"</code> 94's <code>"Abbreviation"</code> is <code>l </code> (trailing space) with <code>"Meaning"</code> of
<code>laugh</code>; <code>"ID"</code> 186's <code>"Abbreviation"</code> is <code>th </code> (trailing space) with
<code>"Meaning"</code> of <code>threaten</code>. Both violate the target's
<code>trimmedofspaces_check</code>. Trimming either value makes it collide with an
existing unpadded row (<code>"ID"</code> 93 <code>l</code> -> <code>locomotion/travel</code>; <code>"ID"</code> 185
<code>th</code> -> <code>throw</code>), folding directly into the Problem #241 duplicate-
abbreviation resolution.

=== Bad Data ===

<pre>
SELECT "ID", "Abbreviation", "Meaning"
FROM clean.tlk_brecord_notes_codes
WHERE "Abbreviation" IS DISTINCT FROM BTRIM("Abbreviation");
</pre>

The dated source returns exactly two rows: <code>94, "l ", laugh</code> and <code>186,
"th ", threaten</code>.

=== Solution ===

Approved by the investigator on 2026-09-19, as an explicit exception to
this repository's usual prohibition on clean-stage mutation: trim
<code>"Abbreviation"</code> in <code>conversion/clean.sql</code>, in its own reviewable
statement, rather than in the loader or projection. An initial
implementation matched the <code>UPDATE</code> by exact <code>"ID" IN (94, 186)</code>; on
review the investigator preferred applying <code>BTRIM</code> unconditionally, as a
general clean-stage normalization, since it produces the identical result
here (no other row is padded) and reads more simply as ordinary cleanup
rather than a two-row special case. The dated source affects exactly the
same two rows either way, <code>"ID"</code> 94 and <code>"ID"</code> 186. After this fix, <code>"ID"</code>
94's abbreviation is <code>l</code> and <code>"ID"</code> 186's is <code>th</code>, each now a member of an
ordinary Problem #241 duplicate group.

Implemented and verified on 2026-09-19. Added an unconditional
<code>UPDATE ... SET "Abbreviation" = BTRIM("Abbreviation")</code> to
<code>conversion/clean.sql</code>, independently reviewable from the loader, with a
comment documenting that the dated source's only two affected rows are
<code>"ID"</code> 94 and <code>"ID"</code> 186. Read-only re-checks against the local
PostgreSQL 18.6 <code>sokwedb</code> database confirmed the two exact pre-fix rows
before the trim, that applying the statement leaves zero padded rows
afterward, and that the two documented rows' post-fix unpadded values
(<code>l</code>, <code>th</code>) are unchanged from what the narrowly-targeted form would have
produced.

== * (#241) TLK_BRECORD_NOTES_CODES duplicate abbreviations after trimming ==

=== Problem ===

After Problem #240's fix, 26 case-insensitive abbreviations occur more than
once, with genuinely different <code>Meaning</code> text recorded for the same
abbreviation depending on note-taking context (for example <code>r</code> means
<code>rest</code>, <code>reassurance</code>, <code>reunion</code>, or <code>resume</code> across four separate rows).
The target permits only one row per case-insensitive abbreviation via a
unique index on <code>lower(normalize(Abbreviation))</code>.

=== Bad Data ===

<pre>
SELECT lower(normalize(BTRIM("Abbreviation"))) AS norm_abbr, count(*)
, array_agg("ID" ORDER BY "ID") AS ids
, array_agg("Meaning" ORDER BY "ID") AS meanings
FROM clean.tlk_brecord_notes_codes
GROUP BY 1
HAVING count(*) > 1
ORDER BY 1;
</pre>

The dated source returns exactly 26 groups covering 55 rows: 24 groups of 2
rows, one group of 3 (<code>ds</code>), and one group of 4 (<code>r</code>).

=== Solution ===

Superseded on 2026-09-19. An initial approval picked the lowest source
<code>"ID"</code> in each group to survive. On review the investigator rejected that
tie-break: choosing a survivor's <code>Meaning</code> over its siblings' is itself an
uncorroborated judgment call about which recorded meaning is "correct",
which is exactly what a targeted, documented exclusion is meant to avoid.

Approved by the investigator on 2026-09-19 (final): exclude every row in
each of the 26 case-insensitive duplicate-abbreviation groups, not only the
non-survivors. None of the 26 ambiguous abbreviations is loaded under any
<code>Meaning</code>; only the 155 rows whose abbreviation is unique in the source
(after Problem #240's trim) are eligible. This drops all 55 grouped rows,
not 29. The excluded rows, and their meanings, remain visible, unmodified,
in <code>clean.tlk_brecord_notes_codes</code>. Do not infer, rank, or merge meanings;
do not pick a survivor by <code>"ID"</code> or any other criterion.

Implemented and verified on 2026-09-19. <code>conversion/tlk_brecord_notes_codes_projection.sql</code>
excludes every row whose case-insensitive abbreviation occurs more than
once in <code>clean.tlk_brecord_notes_codes</code> (after the Problem #240 trim),
proved against an independently-derived 55-row, 26-abbreviation exclusion
set. Read-only sanity (<code>conversion/load_tlk_brecord_notes_codes_sanity.sql</code>)
against the local PostgreSQL 18.6 <code>sokwedb</code> database confirmed the
exclusion set matches exactly and that the canonical projection is exactly
155 rows with 155 distinct case-insensitive abbreviations, none of them one
of the 26 excluded abbreviations.

The direct set-based loader (<code>conversion/load_tlk_brecord_notes_codes.sql</code>)
inserted exactly 155 rows. Bidirectional <code>EXCEPT ALL</code> between the
canonical projection and the loaded rows
(<code>conversion/load_tlk_brecord_notes_codes_validation.sql</code>) returned no
differences in either direction, every loaded abbreviation was proven
unique under <code>lower(normalize(...))</code> and to have exactly one matching
source row, every loaded <code>Meaning</code> was proven to match that single source
row exactly, and none of the 26 excluded abbreviations was proven present
in the loaded rows. An intentional post-insert failure (<code>SELECT 1/0</code>)
rolled the single transaction back to zero target rows. An authorized
<code>TRUNCATE</code> of only the disposable target table followed by an unchanged
sanity/load/validation sequence, run twice, reproduced the identical
sorted two-column payload digest, <code>27da98a74e6378218418652dda914c95</code>, on
both runs. <code>git diff --check</code>, a
local-only Make dry run of the full <code>load_data</code> sequence confirming correct
ordering, a direct Make-driven execution of the new
<code>load_tlk_brecord_notes_codes_sanity</code>/<code>load_tlk_brecord_notes_codes</code>/
<code>load_tlk_brecord_notes_codes_validation</code> targets, and a direct <code>m4</code> render
of <code>doc/src/housekeeping/tlk_brecord_notes_codes.m4</code> all passed without
error.

== * (#242) FOOD_VARIATIONS eligible rows lack a LocalName ==

=== Problem ===

Problem #161 deferred all of <code>clean.food_variations_lookup</code> to a future
conversion because of missing and unresolved <code>LocalName</code> values. Picking
this conversion back up: the target <code>housekeeping.food_variations.LocalName</code>
is <code>NOT NULL</code>, but 224 of the source's 1,575 rows have a NULL
<code>fvl_fl_local_food_name</code>. 40 of those 224 rows have a
<code>fvl_food_spelling_variant</code> that itself exactly matches an existing
<code>codes.food_names</code> entry (for example <code>DONGO</code>, <code>KIFUMUFUMU</code>), but the
other 184 do not, so there is no single reliable substitute value across
all 224 rows.

=== Bad Data ===

<pre>
SELECT count(*) AS null_localname_rows
, count(*) FILTER (
WHERE EXISTS (
SELECT 1 FROM codes.food_names AS fn
WHERE lower(normalize(fn.name)) =
lower(normalize(fvl.fvl_food_spelling_variant))))
AS variant_matches_food_names
FROM clean.food_variations_lookup AS fvl
WHERE fvl.fvl_fl_local_food_name IS NULL;
</pre>

The 2026-09-19 snapshot returns 224 and 40.

=== Solution ===

Approved by the investigator on 2026-09-19: exclude all 224 rows with a
NULL <code>LocalName</code>, including the 40 whose <code>Variant</code> happens to match a
<code>codes.food_names</code> entry. Do not infer a <code>LocalName</code> from the <code>Variant</code>,
even for those 40 rows. This is a straightforward domain predicate
(<code>fvl_fl_local_food_name IS NOT NULL</code>), not an exact-tuple exclusion list,
since excluding by this condition cannot ambiguously over- or
under-exclude the way a duplicate-abbreviation tie-break could. This
leaves at most 1,351 eligible rows before any other approved exclusion.

Implemented and verified on 2026-09-20. <code>conversion/food_variations_projection.sql</code>
excludes every row via the domain predicate <code>fvl_fl_local_food_name IS
NOT NULL</code>. Read-only sanity (<code>conversion/load_food_variations_sanity.sql</code>)
against the local PostgreSQL 18.6 <code>sokwedb</code> database confirmed the exact
224-row NULL-<code>LocalName</code> count and that the canonical projection is
exactly 1,351 rows with 1,351 distinct case-insensitive <code>Variant</code> values.
The direct set-based loader (<code>conversion/load_food_variations.sql</code>)
inserted exactly 1,351 rows. Bidirectional <code>EXCEPT ALL</code> between the
canonical projection and the loaded rows
(<code>conversion/load_food_variations_validation.sql</code>) returned no
differences in either direction, every loaded <code>Variant</code> was proven unique
under <code>lower(normalize(...))</code>, every loaded <code>LocalName</code> was proven to
match its source row exactly, and no loaded row was proven to have a NULL
source <code>LocalName</code>. An intentional post-insert failure (<code>SELECT 1/0</code>)
rolled the single transaction back to zero target rows. An authorized
<code>TRUNCATE</code> of only the disposable target table followed by an unchanged
sanity/load/validation sequence, run twice, reproduced the identical
sorted two-column payload digest, <code>c510323457dcb801412441eda0ceb09c</code>, on
both runs. <code>git diff --check</code>, a local-only Make dry run of the full
<code>load_data</code> sequence confirming correct ordering, a direct Make-driven
execution of the new <code>load_food_variations_sanity</code>/<code>load_food_variations</code>/
<code>load_food_variations_validation</code> targets, and a direct <code>m4</code> render of
<code>doc/src/housekeeping/food_variations.m4</code> all passed without error.

== (#243) FOOD_VARIATIONS orphaned LocalName values are not cross-checked ==

=== Problem ===

Of the 1,351 rows surviving Problem #242, 38 have a <code>LocalName</code> that does
not match any <code>clean.food_lookup</code> entry case-insensitively, and a
stricter 79 do not match the smaller, already-loaded production
<code>codes.food_names</code> table. <code>housekeeping.food_variations</code> declares no
foreign key to either table, so nothing in the schema forces a
resolution; several of the mismatches form typo clusters pointing at a
name that is not itself canonical either (for example eight variants —
<code>MBOBOGOLO</code>, <code>MTOMBOGOLO</code>, <code>MUTBOGORO</code>, <code>MUTOBOGALO</code>, <code>MUTOBOGO</code>,
<code>MUTOBOLO</code>, <code>MUTOBORGORO</code>, <code>MUTOLGURO</code> — all record <code>LocalName</code>
<code>MTOBOGOLO</code>, which appears in neither reference table).

=== Bad Data ===

<pre>
SELECT count(*) FILTER (
WHERE NOT EXISTS (
SELECT 1 FROM clean.food_lookup AS fl
WHERE lower(fl.fl_local_food_name) = lower(fvl.fvl_fl_local_food_name)))
AS orphan_vs_food_lookup
, count(*) FILTER (
WHERE NOT EXISTS (
SELECT 1 FROM codes.food_names AS fn
WHERE lower(normalize(fn.name)) =
lower(normalize(fvl.fvl_fl_local_food_name))))
AS orphan_vs_food_names
FROM clean.food_variations_lookup AS fvl
WHERE fvl.fvl_fl_local_food_name IS NOT NULL;
</pre>

The 2026-09-20 snapshot returns 38 and 79, unchanged since 2026-09-19.

=== Solution ===

Approved by the investigator on 2026-09-20: do not cross-check <code>LocalName</code>
against <code>clean.food_lookup</code>, <code>codes.food_names</code>, or any other table for
this conversion. Load every eligible row's <code>LocalName</code> as free text
exactly as recorded, matching, mismatching, or orphaned alike. Sanity may
report the 38/79 counts as informational profiling, but neither figure
excludes any row. This mapping excludes no additional rows beyond
Problem #242's 224.

Implemented and verified on 2026-09-20. The canonical projection applies
no cross-check against <code>clean.food_lookup</code> or <code>codes.food_names</code>; sanity
reports the 38/79 mismatch counts as <code>RAISE NOTICE</code> profiling only.
Validation confirmed every loaded <code>LocalName</code> matches its source row
exactly regardless of whether it matches either reference table, and the
1,351-row loaded count matches <code>1,575 source - 224 Problem #242
exclusions</code> exactly, confirming no additional row was excluded under this
resolution.

== (#244) REPRO_STATES cannot represent adolescent and not-seen days ==

=== Problem ===

<code>clean.female_reproductive_states.repro_state</code> contains <code>A</code> for adolescent
and SQL NULL for not seen, but production <code>REPRO_STATES.State</code> is <code>NOT NULL</code>
and permits only cycling (<code>C</code>), pregnant (<code>P</code>), and lactating (<code>L</code>). The
dated 2026-09-19 source has 7,582 <code>A</code> rows and 218,370 NULL rows.

=== Bad Data ===

<pre>
SELECT repro_state, count(*)
FROM clean.female_reproductive_states
GROUP BY repro_state
ORDER BY repro_state NULLS LAST;
</pre>

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: adolescent <code>A</code>
was added to the target state domain and <code>REPRO_STATES.State</code> was made
nullable. Map source <code>A</code>, <code>C</code>, <code>P</code>, and <code>LA</code> to target <code>A</code>, <code>C</code>, <code>P</code>, and
<code>L</code>; preserve source NULL as target NULL. Do not carry a previous state
into a not-seen day. The generated SQL and documentation were regenerated;
catalog inspection and a rollback-only NULL-State insert verified the change.

== (#245) REPRO_STATES requires sparse support attributes ==

=== Problem ===

Production <code>Source</code>, <code>EstrusDayCert</code>, and <code>LactEndCert</code> are <code>NOT NULL</code>, but
the dated source contains 646,257, 626,192, and 646,602 NULL values
respectively. The 30 rows having an EstrusDay but NULL EstrusDayCert are
valid under the approved sparse representation.

=== Bad Data ===

<pre>
SELECT count(*) FILTER (WHERE state_change_dec_source IS NULL) AS null_source
, count(*) FILTER (WHERE day_of_est_cert IS NULL) AS null_ed_cert
, count(*) FILTER (WHERE lactat_end_cert IS NULL) AS null_le_cert
, count(*) FILTER (
WHERE day_of_estr IS NOT NULL AND day_of_est_cert IS NULL)
AS estrus_day_without_cert
FROM clean.female_reproductive_states;
</pre>

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: the three
production columns were made nullable. Preserve source NULL as SQL NULL;
do not carry values forward and do not convert NULL to empty text. The
support tables reject empty keys, and an empty key would invent a category
rather than represent missing information. Catalog inspection and a
rollback-only sparse-row insert verified the change.

== (#246) REPRO_STATES support tables are empty ==

=== Problem ===

The four required support tables are empty. The source has twelve distinct
trimmed change-source labels, ED certainty values <code>1</code> through <code>5</code> plus the
Problem #247 typo, LE certainty values <code>1</code> through <code>4</code>, and three parity
values. The Access dump contains no lookup tables defining them.

=== Bad Data ===

<pre>
SELECT upper(btrim(state_change_dec_source)) AS code, count(*)
FROM clean.female_reproductive_states
WHERE state_change_dec_source IS NOT NULL
GROUP BY 1
ORDER BY 1;

SELECT parity, count(*)
FROM clean.female_reproductive_states
GROUP BY parity
ORDER BY parity;
</pre>

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: load uppercase
parity keys <code>IMMATURE</code>, <code>NULLIPAROUS</code>, and <code>PAROUS</code>. Normalize populated
change-source keys with <code>upper(btrim(...))</code>, retaining the twelve semantic
labels and expanding their state prefixes in Description; the exact
key/description pairs are recorded in <code>repro_states_loader_handoff.md</code>.

Load ED keys <code>1</code> through <code>4</code> with the approved descriptions based on the
length of the unobserved gap following the last full swelling. Load LE keys
<code>1</code> through <code>4</code> with the approved minimum/maximum swelling, 14-day, and
first-cycle criteria. ED code <code>5</code> has no support row; see Problem #248, which
now corrects <code>5</code> to <code>4</code> in the clean schema rather than excluding it, so no
row ever carries an unsupported code. No support row represents NULL; see
Problem #245. Sanity verifies bidirectional parity of every support key and
description; the clean retry loaded exactly 12 change sources, four ED
certainties, four LE certainties, and three parities.

== (#247) FEMALE_REPRODUCTIVE_STATES has estrus certainty 22 ==

=== Problem ===

One row, SIF on 2004-03-21, has <code>day_of_est_cert = '22'</code>. The investigator
confirmed that this is a typo for <code>2</code>.

=== Bad Data ===

<pre>
SELECT *
FROM clean.female_reproductive_states
WHERE day_of_est_cert = '22';
</pre>

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: change <code>22</code> to
<code>2</code> in the clean schema. The focused clean-stage execution changed exactly
one row; sanity verifies that <code>22</code> is absent and the corrected <code>2</code> count is
8,251.

== (#248) FEMALE_REPRODUCTIVE_STATES estrus certainty 5 is unsupported ==

=== Problem ===

The dated source has 30 rows with <code>day_of_est_cert = '5'</code>. The investigator
provided meanings for ED certainty codes <code>1</code> through <code>4</code>; code <code>5</code> has no
supported meaning.

=== Bad Data ===

<pre>
SELECT *
FROM clean.female_reproductive_states
WHERE day_of_est_cert = '5'
ORDER BY id, date;
</pre>

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: temporarily
exclude exactly these 30 rows from the reproductive-state projection. Code
<code>5</code> is absent from <code>ED_CERTAINTIES</code>. Sanity verifies both the 30-row class and
that it overlaps none of Problems #251 or #254.

Amended by the investigator on 2026-09-24: convert <code>day_of_est_cert = '5'</code> to
<code>4</code> in the clean schema instead of excluding the 30 rows. No <code>5</code> value is
retained, no row is excluded, and no support row is added for <code>5</code>; the
correction is applied alongside the Problem #247 typo correction in
<code>conversion/clean.sql</code>. Sanity now asserts an exact 2,325-row <code>4</code> count and
rejects any remaining <code>5</code> value as a corrected-domain violation. The
approved projection grew from 649,495 to 649,525 rows, and the pre-expansion
eligible/exclusion counts changed accordingly; the refreshed counts are
verified in <code>conversion/repro_states_profile.sql</code>,
<code>conversion/load_repro_states_sanity.sql</code>, and
<code>conversion/load_repro_states_validation.sql</code>.

== (#249) FEMALE_REPRODUCTIVE_STATES swelling bounds have no target ==

=== Problem ===

Source columns <code>min_sw</code> and <code>max_sw</code> have no <code>REPRO_STATES</code> destination. The
dated source has 160,268 rows where both are populated and 486,621 where both
are NULL.

=== Bad Data ===

<pre>
SELECT min_sw, max_sw, count(*)
FROM clean.female_reproductive_states
GROUP BY min_sw, max_sw
ORDER BY min_sw NULLS LAST, max_sw NULLS LAST;
</pre>

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: do not map or
convert these fields because they are generated by <code>build_swelling_states()</code>.
They remain unchanged in clean as audit evidence and are absent from the
shared production projection. This decision excludes no rows.

== (#250) FEMALE_REPRODUCTIVE_STATES has two incorrect female IDs ==

=== Problem ===

Female IDs <code>OBE</code> and <code>UWE</code> have no production biography match, affecting
1,404 and 29 rows respectively.

=== Bad Data ===

<pre>
SELECT id, count(*)
FROM clean.female_reproductive_states
WHERE id IN ('OBE', 'UWE')
GROUP BY id
ORDER BY id;
</pre>

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: correct <code>OBE</code> to
<code>POR</code> and <code>UWE</code> to <code>MGF</code> in the clean schema. The focused clean-stage
execution changed exactly 1,433 rows; sanity verifies 1,404 <code>POR</code> and 29 <code>MGF</code>
rows, no old labels, and applies study-boundary checks afterward.

== * (#251) FEMALE_REPRODUCTIVE_STATES dates fall outside study intervals ==

=== Problem ===

After applying Problem #250's female corrections, 2,145 dated source rows
fall before EntryDate or after DepartDate. This replaces the earlier
matched-ID-only count of 1,175: correcting <code>OBE</code> to <code>POR</code> adds 677
before-entry and 293 after-departure rows. The other corrected ID, <code>MGF</code>,
adds none.

=== Bad Data ===

<pre>
WITH corrected AS (
SELECT source.*
, CASE source.id
WHEN 'OBE' THEN 'POR'
WHEN 'UWE' THEN 'MGF'
ELSE source.id
END AS animid
FROM clean.female_reproductive_states AS source)
SELECT corrected.*
FROM corrected
JOIN sokwedb.biography_data
ON biography_data.animid = corrected.animid
WHERE corrected.date < biography_data.entrydate
OR corrected.date > biography_data.departdate
ORDER BY corrected.animid, corrected.date;
</pre>

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: temporarily
exclude every corrected-female/date row outside the inclusive study interval.
The shared projection does not alter dates, and sanity independently verifies
exactly 2,145 exclusions and no overlap with Problems #248 or #254.

== (#252) FEMALE_REPRODUCTIVE_STATES compound offspring IDs represent twins ==

=== Problem ===

Five slash-delimited <code>young_kid_id</code> values represent twins rather than invalid
single identifiers. A single YKID foreign key cannot store both offspring.

=== Bad Data ===

<pre>
SELECT young_kid_id, count(*)
FROM clean.female_reproductive_states
WHERE young_kid_id LIKE '%/%'
GROUP BY young_kid_id
ORDER BY young_kid_id;
</pre>

The raw dated counts are <code>GAB2/GAB3</code> 6, <code>GLI/GLD</code> 2,013, <code>GY/GL</code> 296,
<code>SG/CT</code> 3,472, and <code>SHO/ROO</code> 368. After preceding exclusions, 5,546 source
rows are eligible and expand to 11,092 target rows; Problem #251 removes 609
of the <code>SG/CT</code> rows first.

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: split each
eligible compound row into two target rows, one per offspring, copying all
other source payload before deriving YKAge separately for each offspring. The
projection names only the five approved labels and fails sanity on any new
slash-delimited label. It expands 5,546 eligible source rows to 11,092 output
rows. See Problem #258 for the resulting female/date/YKID invariant.

== (#253) FEMALE_REPRODUCTIVE_STATES offspring AMA is incorrect ==

=== Problem ===

The unmatched offspring label <code>AMA</code> occurs on 287 dated source rows. The
investigator confirmed that these values should identify <code>AME</code>.

=== Bad Data ===

<pre>
SELECT *
FROM clean.female_reproductive_states
WHERE young_kid_id = 'AMA'
ORDER BY id, date;
</pre>

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: correct <code>AMA</code> to
<code>AME</code> in the clean schema. The focused clean-stage execution changed exactly
287 rows, and sanity verifies that <code>AMA</code> is absent and 287 rows use <code>AME</code>.
This correction excludes no rows.

== * (#254) FEMALE_REPRODUCTIVE_STATES offspring TTB2 is unsupported ==

=== Problem ===

The unmatched offspring label <code>TTB2</code> occurs on 546 dated source rows and has
no approved production biography identity.

=== Bad Data ===

<pre>
SELECT *
FROM clean.female_reproductive_states
WHERE young_kid_id = 'TTB2'
ORDER BY id, date;
</pre>

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: temporarily
exclude exactly these 546 rows. The projection creates no biography row and
infers no replacement offspring. Sanity verifies the exact class and that it
overlaps neither Problem #248 nor Problem #251. Problem #248's 2026-09-24
amendment converts its 30 rows to a supported code rather than excluding
them, so this class no longer shares an exclusion mechanism with Problem
#248, but the two remain non-overlapping in the source data.

== * (#255) Source youngest-offspring ages disagree with biography ==

=== Problem ===

Of 268,336 originally resolved source rows, 115,527 source
<code>youngest_kid_age</code> values differ from <code>Date - BIOGRAPHY_DATA.BirthDate</code>.
After approved corrections, preceding exclusions, and twin expansion, every
retained YKID has a biography BirthDate, but 219 nonsplit rows derive a
negative age from -41 through -1. Production YKAge must be nonnegative.

=== Bad Data ===

<pre>
WITH expanded AS (
SELECT source.id, source.date
, CASE btrim(part.ykid) WHEN 'AMA' THEN 'AME'
ELSE btrim(part.ykid) END AS ykid
FROM clean.female_reproductive_states AS source
CROSS JOIN LATERAL
regexp_split_to_table(source.young_kid_id, '/') AS part(ykid)
WHERE source.young_kid_id IS NOT NULL
AND source.young_kid_id <> 'TTB2')
SELECT expanded.*, biography_data.birthdate
, expanded.date - biography_data.birthdate AS derived_age
FROM expanded
JOIN sokwedb.biography_data
ON biography_data.animid = expanded.ykid
WHERE expanded.date - biography_data.birthdate < 0
ORDER BY expanded.id, expanded.date, expanded.ykid;
</pre>

The implementation profile must additionally apply Problems #248, #250, and
#251 in projection order and assert the final 219-row count independently.

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: ignore source
<code>youngest_kid_age</code> and derive target YKAge from target Date minus the resolved
offspring's biography BirthDate. The projection excludes the 219 rows whose
derived age is negative. Sanity verifies zero unresolved retained offspring,
and validation checks every loaded age against biography. The original
115,527-value disagreement remains in the reproducible profile.

== * (#256) FEMALE_REPRO_HISTORY is outside reproductive-state scope ==

=== Problem ===

<code>clean.female_repro_history</code> is related historical data but is neither a
duplicate nor a drop-in supplement for <code>clean.female_reproductive_states</code>.
It has 128,306 rows, including 764 female/date keys absent from the requested
daily source, and overlapping fields differ extensively.

=== Bad Data ===

<pre>
SELECT count(*) AS rows
, count(DISTINCT (id, date)) AS female_date_keys
FROM clean.female_repro_history;
</pre>

=== Solution ===

Approved by the investigator on 2026-09-21: do not include
<code>clean.female_repro_history</code> in this conversion. Retain the reconciliation
queries as profile evidence only. This decision does not exclude any row from
the requested source.

== (#257) REPRO_STATES AnimID updates bypass integrity checks ==

=== Problem ===

<code>repro_states_func()</code> checks sex only on INSERT and checks study boundaries
only on INSERT or a Date change. A rollback-only probe changed only AnimID on
a valid female row to male <code>AL</code> on 2004-11-19, after AL's 1999-01-13
DepartDate; the UPDATE was accepted.

=== Bad Data ===

<pre>
SELECT pg_get_functiondef('sokwedb.repro_states_func()'::regprocedure);
</pre>

The returned function gates the sex check on <code>TG_OP = 'INSERT'</code> and the date
check on <code>TG_OP = 'INSERT' OR NEW.date <> OLD.date</code>.

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: the owning M4
reruns female-sex validation whenever AnimID changes and reruns study-boundary
validation whenever AnimID or Date changes. Rollback-only tests verified that
updates to a male AnimID, to a female outside her study interval, and to a date
outside the current female's study interval are rejected.

== (#258) REPRO_STATES lacks the female/date/offspring invariant ==

=== Problem ===

Production permits duplicate reproductive-state rows. The earlier proposed
female/date uniqueness rule is too strict because Problem #252 requires one
row per twin; the intended identity is female, date, and youngest offspring.
Rows with no youngest offspring must still be unique per female/date.

=== Bad Data ===

<pre>
SELECT animid, date, ykid, count(*)
FROM sokwedb.repro_states
GROUP BY animid, date, ykid
HAVING count(*) > 1
ORDER BY animid, date, ykid NULLS FIRST;
</pre>

A rollback-only probe inserted two otherwise valid rows for AND on
2004-11-19 and both were accepted. The dated approved projection has zero
duplicate <code>(AnimID, Date, YKID)</code> keys when NULL YKID is treated as equal.

=== Solution ===

Approved by the investigator and implemented on 2026-09-21: a unique index
was added on <code>(AnimID, Date, YKID) NULLS NOT DISTINCT</code>, and the table
documentation describes one row per female/date/youngest-offspring.
Rollback-only tests verified that duplicate NULL and non-NULL YKID keys are
rejected while two distinct twin YKIDs on one female/date are accepted.

== (#259) ELO_RANKS_DAILY_KK_F has rows outside GK's and LB's recorded community ==

=== Problem ===

<code>clean.elo_ranks_daily___kk_females</code> maps directly to <code>sokwedb.elo_ranks_daily_kk_f</code>,
a table scoped to the <code>KK</code> (Kasekela) community. An inclusive join to
<code>COMM_MEMBS</code> found 821 rows for four females that do not fall within a <code>KK</code>
interval on their ranked date:

* <code>GK</code>, 181 rows, 1972-11-01 through 1973-04-30: 61 days have no <code>COMM_MEMBS</code>
interval at all, and the remaining 120 days fall within an <code>HK</code> (Kahama)
interval.
* <code>LB</code>, 638 rows, 1973-01-01 through 1974-09-30: all 638 days fall within an
<code>HK</code> interval.
* <code>ML</code>, 1 row, 1986-10-24: no <code>COMM_MEMBS</code> interval (see Problem #260).
* <code>AL</code>, 1 row, 1999-01-14, in <code>sokwedb.elo_ranks_daily_kk_m</code>: no <code>COMM_MEMBS</code>
interval (see Problem #260).

This spans the historical <code>KK</code>/<code>HK</code> community split (<code>HK</code> starts 1973-01-01,
ends 1977-12-31 per <code>codes.COMM_IDS</code>). Neither the target schema nor the
current loader constrains ranking dates to a <code>COMM_MEMBS</code> interval, so nothing
in production would reject any treatment of these rows.

=== Bad Data ===

<pre>
SELECT e.date, e.individual
FROM clean.elo_ranks_daily___kk_females e
WHERE NOT EXISTS (
SELECT 1 FROM sokwedb.comm_membs cm
WHERE cm.animid = e.individual
AND e.date BETWEEN cm.startdate AND cm.enddate
AND cm.commid = 'KK'
)
ORDER BY e.date;
</pre>

Returns 820 rows (<code>GK</code>, <code>LB</code>, <code>ML</code>); the equivalent query against
<code>clean.elo_ranks_daily___kk_males</code> returns 1 row (<code>AL</code>).
<code>clean.elo_ranks_daily___mt_females</code> has zero such rows.

=== Solution ===

Approved by the investigator on 2026-09-22: preserve all 821 rows unchanged.
The daily Elo ranking is a precomputed, published output; the conversion's
role is to carry it forward as supplied, not to re-scope it by community
after the fact. No row is excluded and no rank is recalculated. This
decision applies identically to the 758 days recorded in <code>HK</code> and the 63 days
with no <code>COMM_MEMBS</code> interval at all.

== (#260) ELO_RANKS_DAILY ranking dates trail DepartDate by one day for ML and AL ==

=== Problem ===

Two rows are dated exactly one day after the individual's
<code>BIOGRAPHY_DATA.DepartDate</code>:

* <code>ML</code> in <code>clean.elo_ranks_daily___kk_females</code>: ranked 1986-10-24, DepartDate
1986-10-23.
* <code>AL</code> in <code>clean.elo_ranks_daily___kk_males</code>: ranked 1999-01-14, DepartDate
1999-01-13.

Neither row is otherwise malformed, and the target schema does not constrain
ranking dates to an individual's study interval, so both would load without
error. Both rows are also members of the 63 "no <code>COMM_MEMBS</code> interval" rows
described in Problem #259.

=== Bad Data ===

<pre>
SELECT e.date, e.individual, b.departdate
FROM clean.elo_ranks_daily___kk_females e
JOIN sokwedb.biography_data b ON b.animid = e.individual
WHERE e.date = b.departdate + 1;
-- and the equivalent query against clean.elo_ranks_daily___kk_males
</pre>

Returns exactly the <code>ML</code> and <code>AL</code> rows above; no other individual in any of
the three sources is ranked after their own DepartDate.

=== Solution ===

Approved by the investigator on 2026-09-22: preserve both rows unchanged, as
a final recorded ranking day. Because each ranked day is a complete <code>1..N</code>
permutation, excluding either individual's single row would leave that day
looking complete while no longer being one, so no partial-day exclusion is
created for either row. This decision was made together with, and is
consistent with, Problem #259's community-membership disposition, without
treating the two as the same underlying question.

== (#261) ELO_RANKS_DAILY documentation misstates the ExpNumBeaten formula ==

=== Problem ===

The shared documentation macro (<code>doc/include/elo_ranks_daily_macro.m4</code>)
defines <code>ExpNumBeaten</code> as <code>EloCardinal</code> multiplied by the number of
individuals ranked that day. Every row in all three sources contradicts that
formula.

=== Bad Data ===

<pre>
SELECT e.date, e.individual, e.expnumbeaten, e.elocardinal, d.n,
e.elocardinal * d.n AS documented_formula,
e.elocardinal * (d.n - 1) AS observed_formula
FROM clean.elo_ranks_daily___kk_females e
JOIN (SELECT date, count(*) AS n
FROM clean.elo_ranks_daily___kk_females
GROUP BY date) d ON d.date = e.date
ORDER BY abs(e.expnumbeaten - e.elocardinal * d.n) DESC
LIMIT 5;
</pre>

Every row instead matches <code>EloCardinal * (N - 1)</code>, where <code>N</code> is the number of
individuals ranked that day, within 0.00000006 across all three sources --
i.e. <code>ExpNumBeaten</code> compares each individual against every other ranked
individual, not against themselves.

=== Solution ===

Approved by the investigator on 2026-09-22: correct
<code>doc/include/elo_ranks_daily_macro.m4</code> to describe <code>ExpNumBeaten</code> as
<code>EloCardinal * (N - 1)</code>. Source <code>ExpNumBeaten</code> values are copied into
production unchanged regardless; only the written definition changes, and
this excludes no rows.

== (#262) Shared ELO_RANKS_DAILY index macros target only KK_F ==

=== Problem ===

The shared <code>elo_ranks_daily_indexes_create</code> and
<code>elo_ranks_daily_indexes_drop</code> macros ignored their sex and community
arguments and hard-coded every index name and target relation to
<code>ELO_RANKS_DAILY_KK_F</code>. Consequently, KK_F had its documented unique and
secondary indexes, while KK_M, MT_F, and MT_M had only their primary-key
indexes. In particular, production did not enforce one row per
<code>(Date, Individual)</code> on those three tables.

=== Bad Data ===

<pre>
SELECT tablename, indexname, indexdef
FROM pg_indexes
WHERE schemaname = 'sokwedb'
AND tablename LIKE 'elo_ranks_daily_%'
ORDER BY tablename, indexname;
</pre>

Before the repair this returned nine indexes for KK_F and only the primary
key for each of KK_M, MT_F, and MT_M. A rollback-only probe confirmed that
KK_M accepted a duplicate <code>(Date, Individual)</code> row.

=== Solution ===

Implemented on 2026-09-23: both owning macros now derive the relation and
index names from their sex and community arguments. All four generated
create/drop pairs were reviewed independently and installed, leaving each
table with its primary key plus the expected unique and seven secondary
indexes. Rollback-only probes verified that duplicate INSERT and UPDATE
collisions are rejected in KK_F, KK_M, MT_F, and MT_M, and that all probe
rows are removed by rollback.

== (#263) BIOGRAPHY_DATA reverse sex validation omits three ELO_RANKS_DAILY tables ==

=== Problem ===

Each ELO_RANKS_DAILY table's own trigger requires its Individual to have the
sex encoded by the table suffix. The reverse <code>BIOGRAPHY_DATA</code> trigger,
however, checked only <code>ELO_RANKS_DAILY_KK_F</code>. After a valid row was inserted,
changing the referenced biography sex could therefore leave KK_M, MT_F, or
MT_M in a state their own insert/update trigger would reject.

=== Bad Data ===

The defect was reproduced with rollback-only female and male biography
fixtures. After inserting one valid rank row in turn, the following updates
were accepted for MT_F, KK_M, and MT_M, while the equivalent KK_F update was
rejected:

<pre>
UPDATE sokwedb.biography_data
SET sex = 'M'
WHERE animid = 'female fixture';

UPDATE sokwedb.biography_data
SET sex = 'F'
WHERE animid = 'male fixture';
</pre>

=== Solution ===

Implemented on 2026-09-23: the owning biography trigger M4 now uses one
parameterized reverse-check fragment for KK_F, KK_M, MT_F, and MT_M. Focused
rollback-only tests verified that female-to-male changes are rejected when
the individual is referenced by KK_F or MT_F, male-to-female changes are
rejected when referenced by KK_M or MT_M, and no fixture rows remain.

== * (#264) BIOGRAPHY_DATA REPRO_STATES diagnostic can produce a NULL RAISE option ==

=== Problem ===

While selecting an existing female for the Problem #263 rollback probe,
changing <code>AND</code> from female to male reached the intended reverse REPRO_STATES
integrity check but failed while constructing its diagnostic. Several
REPRO_STATES fields included in <code>DETAIL</code> are nullable and are concatenated
without <code>textualize()</code>. A NULL field makes the entire <code>DETAIL</code> expression
NULL, which PL/pgSQL rejects before it can raise the intended
<code>integrity_constraint_violation</code> message.

=== Bad Data ===

<pre>
BEGIN;
UPDATE sokwedb.biography_data
SET sex = 'M'
WHERE animid = 'AND';
ROLLBACK;
</pre>

This rollback-only probe fails with SQLSTATE <code>22004</code>, <code>RAISE statement option
cannot be null</code>, in <code>lib.biography_data_func()</code> instead of reporting the
referencing REPRO_STATES row.

=== Solution ===

Unresolved and outside the Elo loader scope. Use <code>textualize()</code> or otherwise
NULL-safe formatting for every nullable REPRO_STATES value included in the
trigger diagnostic. The Elo reverse-trigger test uses isolated biography
fixtures so this pre-existing error path does not mask Problem #263.

== * (#265) BIOGRAPHY_DATA ARRIVALS diagnostic references undeclared a_pid ==

=== Problem ===

While selecting an existing male for the Problem #263 rollback probe,
changing <code>AL</code> from male to female reached the ARRIVALS check for a female
assigned the male swelling code. The block's diagnostic references <code>a_pid</code>,
but the block neither declares that variable nor selects <code>roles.pid</code> into it.
PostgreSQL therefore fails while parsing the intended diagnostic expression.

=== Bad Data ===

<pre>
BEGIN;
UPDATE sokwedb.biography_data
SET sex = 'F'
WHERE animid = 'AL';
ROLLBACK;
</pre>

This rollback-only probe fails with SQLSTATE <code>42703</code>, <code>column "a_pid" does not
exist</code>, in <code>lib.biography_data_func()</code> instead of reporting the referencing
ARRIVALS row.

=== Solution ===

Unresolved and outside the Elo loader scope. Declare <code>a_pid</code>, select
<code>roles.pid</code> into it in each applicable ARRIVALS query, and retain it in the
diagnostic. The Elo reverse-trigger test uses isolated biography fixtures so
this pre-existing error path does not mask Problem #263.

== * (#266) FERTILITY_FUNC's overlap check misses contained periods and skips revalidation on Study/AnimID changes ==

=== Problem ===

This is not a data conversion problem. It is a pre-existing defect in
production trigger logic, discovered incidentally while working on
FERTILITY-related conversion problems (see Problems #155 through #158). The
conversion-issues log has no better venue for it, so it is recorded here
rather than left undocumented.

<code>lib.fertility_func()</code>, defined in
<code>db/schemas/lib/triggers/create/fertility.m4</code>, enforces that, per
<code>Study</code> per <code>AnimID</code>, <code>FERTILITY.StartDate</code>/<code>FERTILITY.StopDate</code>
periods must not overlap. The enforcement has two independent gaps.

First, the overlap predicate only tests whether one of <code>NEW</code>'s own
endpoints falls inside an existing row's interval:

<pre>
SELECT fertility.id, fertility.study, fertility.animid
, fertility.startdate, fertility.starttype
, fertility.stopdate, fertility.stoptype
INTO a_id , a_study , a_animid
, a_startdate , a_starttype
, a_stopdate , a_stoptype
FROM fertility
WHERE fertility.id <> NEW.id
AND fertility.study = NEW.study
AND fertility.animid = NEW.animid
AND ((fertility.startdate <= NEW.startdate
AND fertility.stopdate >= NEW.startdate)
OR (fertility.startdate <= NEW.stopdate
AND fertility.stopdate >= NEW.stopdate))
-- Produce a consistent error message
ORDER BY fertility.id;
</pre>

This misses the case where <code>NEW</code>'s period fully contains an existing,
narrower period: neither of <code>NEW.startdate</code>/<code>NEW.stopdate</code> then falls
inside the existing row, so neither disjunct is true and no overlap is
reported. For example, an existing row with <code>StartDate = 2000-03-01</code>,
<code>StopDate = 2000-03-31</code> does not conflict, by this test, with a new row
<code>StartDate = 2000-01-01</code>, <code>StopDate = 2000-12-31</code>, even though the second
period entirely contains the first.

Second, the trigger only re-runs this check on <code>UPDATE</code> when
<code>StartDate</code> or <code>StopDate</code> changes:

<pre>
IF TG_OP = 'INSERT'
OR (NEW.startdate <> OLD.startdate
OR NEW.stopdate <> OLD.stopdate) THEN
-- Periods of fertility cannot overlap.
</pre>

An <code>UPDATE</code> that changes <code>Study</code> or <code>AnimID</code> instead -- moving a
period to a different study or a different female -- skips the check
entirely, even though the row now belongs to a different (<code>Study</code>,
<code>AnimID</code>) partition that was never checked for overlap against the row's
new period. The comparisons are also NULL-unsafe: <code><></code> evaluates to
<code>NULL</code>, not <code>TRUE</code>, when either side is <code>NULL</code>, so a change into or
out of a <code>NULL</code> <code>StartDate</code>/<code>StopDate</code> would likewise fail to
trigger revalidation.

A related, cosmetic defect sits in the same code: the <code>DETAIL</code> error
message loses a closing parenthesis after <code>StopType</code>, so the message
reads <code>...Value (StopType) = (FOO: Overlapping row has...</code> instead of
matching the <code>): Value (...)</code> pattern used for every other field:

<pre>
|| NEW.stoptype
|| ': Overlapping row has Key (FERTILITY.ID) = ('
</pre>

<code>doc/src/analyzed/fertility.m4</code> documents the same incomplete rule the
code implements, and would need to be corrected alongside any fix:

<pre>
For any given female, for any given study, the periods of fertility
cannot overlap.
This means that, per female, per study, the |FERTILITY.StopDate| of
any given row cannot be between, inclusive, the |FERTILITY.StartDate|
and the |FERTILITY.StopDate| of any other row.
</pre>

This prose has the same gap as the code: it defines overlap only in terms of
one row's <code>StopDate</code> falling within another row's range, so it does not
describe the containment case either.

=== Solution ===

Not addressed at this time. A candidate fix was prepared, verified against
the standard inclusive-interval-overlap test, but never committed, and is
recorded here for whoever picks this up next.

In <code>db/schemas/lib/triggers/create/fertility.m4</code>, replace the overlap
predicate with the standard two-sided inclusive-interval test, which covers
containment and shared endpoints in addition to the cases the original
predicate already caught:

<pre>
AND fertility.startdate <= NEW.stopdate
AND NEW.startdate <= fertility.stopdate
</pre>

Broaden the <code>UPDATE</code> revalidation guard to cover every column the
overlap check depends on, using <code>IS DISTINCT FROM</code> so a change into or
out of <code>NULL</code> is not silently ignored:

<pre>
IF TG_OP = 'INSERT'
OR NEW.study IS DISTINCT FROM OLD.study
OR NEW.animid IS DISTINCT FROM OLD.animid
OR NEW.startdate IS DISTINCT FROM OLD.startdate
OR NEW.stopdate IS DISTINCT FROM OLD.stopdate THEN
-- Periods of fertility cannot overlap.
</pre>

Restore the missing closing parenthesis in the <code>DETAIL</code> message so it
reads <code>): Overlapping row has...</code>, matching every other field in the
same message.

In <code>doc/src/analyzed/fertility.m4</code>, correct the prose definition of the
overlap rule to state the actual (fixed) rule, including the containment and
shared-endpoint cases:

<pre>
For any given female, for any given study, the periods of fertility
cannot overlap, endpoints included.
Two rows overlap when the |FERTILITY.StartDate| of each row is on or
before the |FERTILITY.StopDate| of the other row.
This includes periods that share an endpoint and periods in which one
period contains the other.
</pre>

Before committing, confirm the corrected predicate does not reject any
period already present in the loaded production data; if it does, those
rows are genuine, previously-undetected overlaps in the source and must be
investigated on their own terms rather than folded into this fix.

Conversion Data Issues

2026-09-26T00:20:57Z

StevanEarl: Adding problems #137 through #265

Conversion Data Issues

2026-09-26T00:19:36Z

StevanEarl: Update details of Problem #112

Conversion Data Issues

2026-09-26T00:18:33Z

StevanEarl: Update details of Problem #106

Conversion Data Issues

2026-09-26T00:15:49Z

StevanEarl: Update details of Problem #93

Conversion Data Issues

2026-09-26T00:13:44Z

StevanEarl: Update details of Problem #89

Conversion Data Issues

2026-09-26T00:12:30Z

StevanEarl: Update details of Problem #87