diff --git a/02a-Load-Chapters-to-Working-Table.py b/02-Load-Chapters-to-Working-Table.py similarity index 100% rename from 02a-Load-Chapters-to-Working-Table.py rename to 02-Load-Chapters-to-Working-Table.py diff --git a/02b-Extract-Tags-From-Stories.py b/03-Extract-Tags-From-Stories.py similarity index 100% rename from 02b-Extract-Tags-From-Stories.py rename to 03-Extract-Tags-From-Stories.py diff --git a/03-Export-Tags-Authors-Stories.py b/04-Export-Tags-Authors-Stories.py similarity index 100% rename from 03-Export-Tags-Authors-Stories.py rename to 04-Export-Tags-Authors-Stories.py diff --git a/08-Check-ODAP-Tables.py b/06-Check-ODAP-Tables.py similarity index 100% rename from 08-Check-ODAP-Tables.py rename to 06-Check-ODAP-Tables.py diff --git a/04-Rename-Tags.py b/07-Rename-Tags.py similarity index 100% rename from 04-Rename-Tags.py rename to 07-Rename-Tags.py diff --git a/06-Update-Tags-In-Story-Table.py b/08-Update-Tags-In-Story-Table.py similarity index 100% rename from 06-Update-Tags-In-Story-Table.py rename to 08-Update-Tags-In-Story-Table.py diff --git a/README.md b/README.md index 92eebe3..fc6061b 100755 --- a/README.md +++ b/README.md @@ -42,19 +42,18 @@ the file if it is needed for a given stage. ## High-level overview of each step - 01 - Load the original data into a temporary database for processing (AA only) -- 01.5 - Load chapters into the chapters table of the temporary database (AA and custom only) -- 02 - Extract tags from stories and story links into a new `tags` table. (AA and custom only) -- 03 - Export the distinct tags into a spreadsheet to enable wranglers to map tags to AO3 tags, and export story and -story link information into spreadsheets used for searching. (all) -- 04 - Map the tags in the `tags` table to AO3 tags suggested by wranglers. (all) +- 02 - Load chapters into the chapters table of the temporary database (AA and custom only) +- 03 - Extract tags from stories and story links into a new `tags` table. (AA and custom only) +- 04 - Export the distinct tags into a spreadsheet to enable wranglers to map tags to AO3 tags, and export story and story link information into spreadsheets used for searching. (all) - 05 - Create the final tables that will be used for the temp site and copy all the authors, stories and story links. (all) -- 06 - Copy the AO3 tags into the final story and story link rows. (all) -- 08 - Audit the final tables to find common problems. (all) +- 06 - Check the final tables to find common problems. (all) +- 07 - Map the tags in the `tags` table to AO3 tags suggested by wranglers. (all) +- 08 - Copy the AO3 tags into the final story and story link rows. (all) At this point, the final database is ready to be loaded into a [temporary website](https://github.com/otwcode/open-doors-temp-site) that will be used to feed the works into the Archive using its mass import API. -### Step 01 - Load the original database into MySQL (AA and custom) +### Step 01 - Load the original database into MySQL (AA) This step populates a set of temporary tables which will be used for later processes. @@ -62,11 +61,11 @@ This step populates a set of temporary tables which will be used for later proce python 01-Load-Automated-Archive-into-Mysql.py -p .yml -This imports the Automated Archive ARCHIVE_DB.pl file specified in the `db_input_file` property into the +This imports the Automated Archive `ARCHIVE_DB.pl` file specified in the `db_input_file` property into the `temp_db_database` on your MySQL server using the `db_host`, `db_user` and `db_password` properties. The destination database will be destroyed and recreated every time you run the script, so you can safely rerun it as often as needed. -*Import problems*: Some ARCHIVE_DB.pl files contain formatting that breaks the import. Common problems include, but are not limited to: +*Import problems*: Some `ARCHIVE_DB.pl` files contain formatting that breaks the import. Common problems include, but are not limited to: - unescaped single quotes - irregular indents - line breaks within fields @@ -74,37 +73,37 @@ database will be destroyed and recreated every time you run the script, so you c - HTML entities You will get a Python error when something breaks the import; this will include a snippet of the record that could not -be processed, so use that to find the record in the ARCHIVE_DB.pl and look for problems like the above. You will have +be processed, so use that to find the record in the `ARCHIVE_DB.pl` and look for problems like the above. You will have to manually edit the file in your text editor to resolve these issues. *Fields*: All the field names found in the file will be listed in the console output when you run this step, allowing you -to populate the tag fields (see below) with all the relevant fields in the ARCHIVE_DB.pl file. +to populate the tag fields (see below) with all the relevant fields in the `ARCHIVE_DB.pl` file. *Tag fields*: As the metadata in AA files is customisable, you can use the `tag_fields`, `character_fields`, -`relationship_fields` and `fandom_fields` properties to map fields in the ARCHIVE_DB.pl to the right tag columns in the +`relationship_fields` and `fandom_fields` properties to map fields in the `ARCHIVE_DB.pl` to the right tag columns in the temporary database table. ##### Custom archives -The step 01 script can't be used with archives which do not use Automated Archive or eFiction. The metadata for custom -archives needs to be loaded manually or using custom scripts into `authors`, `story_links`, `chapters` -and `stories` tables matching [the Open Doors table schema](shared_python/create-open-doors-tables.sql) in the +The step 01 script can't be used with archives which do not use Automated Archive, and eFiction is handled separately. +The metadata for custom archives needs to be loaded manually or using custom scripts into `authors`, `story_links`, +`chapters` and `stories` tables matching [the Open Doors table schema](shared_python/create-open-doors-tables.sql) in the `temp_db_database`. -### Step 02a - Load Chapters into the chapters table [NEEDS REVISION] +### Step 02 - Load Chapters into the chapters table - python 02-Load-Chapters-to-Open-Doors-Table.py -p .yml + python 02-Load-Chapters-to-Working-Table.py -p .yml -Loads the chapter contents in the output database (this can be run at any point after stage 05). It does this by going -through all the files in the `chapters_path` and trying to find an entry in the chapters table that has the same -`url`. It then copied the contents of the file into the `text` column for that row. +Loads the chapter contents in the output database. It does this by going through all the files in the `chapters_path` +and trying to find an entry in the chapters table that has the same `url`. It then copies the contents of the file into +the `text` column for that row. You will be prompted to answer two questions: Chapter file names are chapter ids? Y/N -Look at the file names in `chapters_path` and compare against the `chapterid` column in the database. For AA or other databases they probably are more likely to be a human -readable name instead (N). +Look at the file names in `chapters_path` and compare against the `chapterid` column in the database. For AA or other +databases, they probably are more likely to be a human readable name instead (N). Importing chapters: pick character encoding (check for curly quotes): 1 = Windows 1252 @@ -116,9 +115,9 @@ If there are duplicate chapters (for example if co-authored stories were listed try to deduplicate them by only keeping the duplicate whose `author_id` is the same as the `author_id` in the `story` table. It will list duplicates it has found in the console output. -### Step 02b - Extract tags from the original stories (AA and custom) +### Step 03 - Extract tags from the original stories (AA and custom) - python 02-Extract-Tags-From-Stories.py -p .yml + python 03-Extract-Tags-From-Stories.py -p .yml This script creates a table called `tags` in the temporary database and denormalises all the tags for every story and story link. This table is the basis for the Tag Wrangling sheet and is used to map the tags back to the story when the final @@ -132,29 +131,20 @@ For multi-fandom archives that specify the fandoms for each story, the `fields_w specify that tags from the listed columns should be exported with the fandom. -### Step 03 - Export tags, authors and stories (all: eFiction, Automated Archive and custom) +### Step 04 - Export tags, authors and stories (all: eFiction, Automated Archive and custom) - python 03-Export-Tags-Authors-Stories.py -p .yml + python 04-Export-Tags-Authors-Stories.py -p .yml Before an external archive can be imported, the following two steps need to be performed: 1. the external archive's tags have to be mapped to the Archive's existing tags. This saves tag wranglers from having to map the external tags manually to the correct ones. -2. all the stories from the original archive have to searched for on the Archive to prevent importing duplicates. +2. all the stories from the original archive have to be searched for on the Archive to prevent importing duplicates. This step exports those two CSV files which you then have to import into Google Spreadsheet and share with the rest of the Open Doors committee. -### Step 04 - Reimport the Tag Wrangling sheet and map the original tags to the new AO3 tags - - python 04-Rename-Tags.py -p .yml - -When Tag Wrangling have finished mapping the tags in Google Drive, export the Google spreadsheet as a CSV file and make -sure its path is specified in `tag_input_file`. This script then copies the AO3 tags from that file into the -`tags` table in the temporary database. - - ### Step 05 - Create the Open Doors tables python 05-Create-Open-Doors-Tables.py -p .yml @@ -170,29 +160,16 @@ until you are sure you are finished with this stage. *Notes*: - The `stories` and `bookmarks` tables will not contain any tags at all after this stage. These aren't added -until you run step 06. -- The `chapters` table will not contain the story contents, which are loaded in step 07. - - -### Step 06 - Copy AO3 tags into the stories table +until you run step 07. - python 06-Update-Tags-In-Story-Table.py -p .yml -This script matches up the AO3 tags from the `tags` table with the corresponding stories. Note -that unlike the other scripts, this one does not destroy any databases or tables, though it does overwrite the tag -fields in the `stories` or `bookmarks` databases. +### Step 06 - Audit final tables for common problems -*Notes*: -- The output for this command (eg "Getting all tags per story...429/429 stories") will report the number of stories in -the tag table, which may be more than the number of stories you have after removing DNI in the previous stage. - -### Step 08 - Audit final tables for common problems - -| python 08-Check-ODAP-Tables.py -p .yml + python 06-Check-ODAP-Tables.py -p .yml This script performs checks on common reasons for Archive rejection, including checking for chapters that are too long, stories that have too many chapters, etc. It makes no attempt to fix problems that arise; you must do that manually. Also note, it will only perform each check (aligned to the old JIRA tickets O, P, Q, R, and S) if no problems have been found on previous steps. If you encounter errors in any step, you will need to repeat running the script and fixing the errors until it exits cleanly. -### Common problems to look out for when processing chapters +#### Common problems to look out for when processing chapters *Tip*: Some of these problems might be easier to fix by loading the chapters into MySQL and then exporting the `chapters` table as a MySQL dump. You can then perform edit-replace operations on all the chapters at once (though be very careful @@ -218,6 +195,27 @@ for the second author, amend the first author, then put the second author ID int - The author table may need to be de-duped +### Step 07 - Reimport the Tag Wrangling sheet and map the original tags to the new AO3 tags + + python 07-Rename-Tags.py -p .yml + +When Tag Wrangling have finished mapping the tags in Google Drive, export the Google spreadsheet as a CSV file and make +sure its path is specified in `tag_input_file`. This script then copies the AO3 tags from that file into the +`tags` table in the temporary database. + +### Step 08 - Copy AO3 tags into the stories table + + python 08-Update-Tags-In-Story-Table.py -p .yml + +This script matches up the AO3 tags from the `tags` table with the corresponding stories. Note +that unlike the other scripts, this one does not destroy any databases or tables, though it does overwrite the tag +fields in the `stories` or `bookmarks` databases. + +*Notes*: +- The output for this command (eg "Getting all tags per story...429/429 stories") will report the number of stories in +the `tags` table, which may be more than the number of stories you have after removing DNI in a previous stage. + + ## Other Scripts ### Remove emails from Open Doors tables @@ -229,14 +227,14 @@ Scans the chapter for emails and removes some of them: - If the email is something different a prompt is shown to the user, for example: ``` - -

Lily, the Vampire Slayer

- -

by: Br@nw3n

- -

Prologue

- -

"Ernie!" Lily Evans + +

Lily, the Vampire Slayer

+ +

by: Br@nw3n

+ +

Prologue

+ +

"Ernie!" Lily Evans Br@nw3n ([W]hitelist, [B]lacklist) ([A]ddress, [D]omain) [C]ontext [R]ewrite domain > ``` @@ -294,3 +292,6 @@ command line using the flags specified. For example: python 01-Load-into-Mysql.py -dh localhost -du root -dd od_dsa -a AA -i /Users/me/Documents/ARCHIVE_DB.pl You will be prompted for any missing settings when you run each stage. + + +