Repository navigation
Conversation
28485ee to
565498f
Compare
3cbeb68 to
153157f
Compare
565498f to
1bef7c1
Compare
8bb44d5 to
42998d1
Compare
32391a2 to
31972d9
Compare
42998d1 to
8409439
Compare
31972d9 to
b082d9e
Compare
8409439 to
f0f52b3
Compare
d502579 to
9f380d6
Compare
| @@ -0,0 +1,4 @@ | |||
| ALTER TABLE ibexa_content_field | |||
There was a problem hiding this comment.
With foreign_key_checks on, MySQL adds a foreign key with ALGORITHM=COPY only: 13 table copies with writes blocked, ibexa_content_field and ibexa_search_object_word_link included. How is it going to behave on MySQL / MariaDB / PostgreSQL with a bigger dataset? CI executes the SQLite files only.
There was a problem hiding this comment.
I've measured it locally on 300k content items, with 3M fields (~1.4–1.9 GB) and 10M word links (~1.8–2 GB), default 128 MB buffers. Total time of all 13 migrations:
| As is | foreign_key_checks = 0 |
|
|---|---|---|
| MySQL 8.0 | 353 s | 15 s |
| MariaDB 10.11 | 135 s | 18 s |
| PostgreSQL 16 | 6 s | – |
Almost all of the MySQL/MariaDB time is the table copy: 304 s (MySQL) and 89 s (MariaDB) for ibexa_search_object_word_link alone, and ~25 s for ibexa_content_field. With the checks on, both reject ALGORITHM=INPLACE ("Adding foreign keys needs foreign_key_checks=OFF"). With them off, both accept ALGORITHM=INPLACE, LOCK=NONE, so the index build that's left doesn't block writes either. PostgreSQL validates each key in a single scan (0.5 s at most here), and NOT VALID + VALIDATE CONSTRAINT didn't change anything, so I'd leave it as it is.
The time grows with the table size, so a site with 10x the word links would spend close to an hour copying tables as it is now. I'd switch foreign_key_checks off around the ALTERs on MySQL/MariaDB, the same way #964 now switches PRAGMA foreign_keys off on SQLite. MySQL doesn't validate the existing rows then, but the migration checks for missing language ids right before, so only rows written during the upgrade itself could slip through.
Should we go with it? @alongosz, what's your take?
There was a problem hiding this comment.
The time grows with the table size, so a site with 10x the word links would spend close to an hour copying tables as it is now. I'd switch
foreign_key_checksoff around theALTERs on MySQL/MariaDB (...)Should we go with it? @alongosz, what's your take?
Makes sense
| @@ -0,0 +1,3 @@ | |||
| ALTER TABLE ibexa_content_type_field_definition_ml | |||
There was a problem hiding this comment.
No index in this variant? If the old key is missing, the ibexa_content_type_field_definition_ml_lang_fk index from schema.yaml may be missing as well. Same for the MySQL one.
There was a problem hiding this comment.
Added in d46cb6a: when no index on the table starts with language_id, the migration creates ibexa_content_type_field_definition_ml_lang_fk before adding the key, in both variants. It looks for any such index rather than that name, since sites upgraded with the installer's 5.0 script still have it as ezcontentclass_attribute_ml_lang_fk.
|
|
||
| if ($this->isSqlite()) { | ||
| $this->abortIf( | ||
| (bool)$this->connection->fetchOne('PRAGMA foreign_keys'), |
There was a problem hiding this comment.
Rebuilding the table is how SQLite itself recommends making changes ALTER TABLE can't: https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes. I've dropped the abort in 9997c72, too: on SQLite the migrations now run outside a transaction, so they can switch PRAGMA foreign_keys off for the rebuild and back on afterwards, as that procedure does. The definitions don't have to follow schema.yaml later on either. Each one is the table as of this migration, and later changes come as new migrations. We only use SQLite for testing, so it's best-effort anyway, and I'd keep it.
| name: { type: string, nullable: false, length: 45, options: { default: '' } } | ||
| foreignKeys: | ||
| ibexa_object_state_group_language_language_fk: { fields: [language_id], foreignTable: ibexa_content_language, foreignFields: [id], options: { onDelete: RESTRICT, onUpdate: CASCADE } } | ||
| ibexa_object_state_group_language_real_language_fk: { fields: [real_language_id], foreignTable: ibexa_content_language, foreignFields: [id], options: { onDelete: RESTRICT, onUpdate: CASCADE } } |
There was a problem hiding this comment.
canDeleteLanguage() checks only language_id for this table ($columns[0]). Add real_language_id to MULTILINGUAL_TABLES_COLUMNS?
There was a problem hiding this comment.
Added in 989b490. canDeleteLanguage() also checked only the first column listed for each table, so it goes through all of them now.
| data_json: { type: text, nullable: true, length: 65535 } | ||
| foreignKeys: | ||
| ibexa_content_type_field_definition_ml_lang_fk: { fields: [language_id], foreignTable: ibexa_content_language, foreignFields: [id], options: { onDelete: CASCADE, onUpdate: CASCADE } } | ||
| ibexa_content_type_field_definition_ml_language_fk: { fields: [language_id], foreignTable: ibexa_content_language, foreignFields: [id], options: { onDelete: RESTRICT, onUpdate: CASCADE } } |
There was a problem hiding this comment.
Why the rename? The 5.0 upgrade script just gave it the _ml_lang_fk name, and the index above keeps it.
There was a problem hiding this comment.
It's not only the name: the key goes from ON DELETE CASCADE to RESTRICT, as the other 12 do. Neither MySQL nor PostgreSQL can change that in place, so it's dropped and re-added either way, and the new name comes for free with that. It lines it up with the other <table>_language_fk keys, and the "already added" check works the same for all 13. RESTRICT matches what core already does, too: canDeleteLanguage() refuses to delete a language this table still uses, so the CASCADE could only kick in on a direct SQL delete.
| status: { type: integer, nullable: true, options: { default: '0' } } | ||
| is_hidden: { type: boolean, nullable: false, options: { default: '0' } } | ||
| foreignKeys: | ||
| ibexa_content_initial_language_fk: { fields: [initial_language_id], foreignTable: ibexa_content_language, foreignFields: [id], options: { onDelete: RESTRICT, onUpdate: CASCADE } } |
There was a problem hiding this comment.
The default 0 can never pass this key, the fixtures here needed languages and initial_language_id added. Are consumers (corporate-account, mcp, measurement, data-intelligence-layer) ready?
There was a problem hiding this comment.
They should be. corporate-account's and mcp's fixtures set initial_language_id (2, and 8 in mcp) on every content and version row and go through FixtureImporter, and measurement doesn't seed any content. data-intelligence-layer sets initial_language_id on its content inserts since ibexa/data-intelligence-layer#63, though its CI ran against #946, not this PR yet.
| * | ||
| * @return array<string, mixed> | ||
| */ | ||
| private function splitLegacySearchIndexLanguageMask( |
There was a problem hiding this comment.
full_dump.php got language_id on all its rows, test_data.yaml keeps language_mask: 3 and goes through this. Why not fix that fixture as well?
There was a problem hiding this comment.
full_dump.php had nothing to convert: its search index rows are older than both language_mask and language_id, so they'd all have landed on language 0. test_data.yaml has the mask, and FixtureImporter splits it like the other masks in that file. The kernel suite reads test-core's own test_data.yaml anyway, which still has the mask, so the conversion has to stay either way.
f0f52b3 to
bcd1688
Compare
9f380d6 to
86e1c98
Compare
…guage table The initial language of content, versions and content types, the language of fields, content names, content type names and object state translations, and the default language of object states and groups now reference ibexa_content_language, ON DELETE RESTRICT ON UPDATE CASCADE like the translation tables. The field definition translations' foreign key, ON DELETE CASCADE until now, follows suit under a new name. AddLanguageForeignKeysMigration adds them after the bitmask migrations, refusing while any row holds a language id without a language. SQLite rebuilds the tables. schema.yaml lists the language table first and the clean data inserts it first; FixtureImporter inserts it first and detaches every row referencing it when a fixture replaces the languages; fixtures and tests use language ids that exist.
…gn key MySQL and MariaDB can't roll back schema changes, so a failure halfway through the single migration left some of the foreign keys added and others not. Each foreign key now has its own migration, which adds it together with its index in one statement and skips itself when the key is already there, so the migrations' status shows which ones are in.
…he language table
…nition translations when it's missing
… id before deleting a language
…nguage foreign keys test
…ort test with TestWith attributes
…adding their foreign key
…key rebuilds The migrations run outside a transaction on SQLite, so PRAGMA foreign_keys can be switched off around the table rebuild, and back on afterwards, instead of aborting when the connection enforces foreign keys.
9997c72 to
b6a64be
Compare
bcd1688 to
7f0bb8d
Compare
|



Note
Stacked on #946, the last part of the language bitmask removal. #965 goes on top of it.
Related PRs:
Description:
Only #941's join tables and the field definition translations had a foreign key to
ibexa_content_language. Now every core column holding a language id does: the initial language of content, versions and content types, the language of fields, content names, content type names and object state translations, the default language of object states and groups, and the legacy search index (ibexa_search_object_word_link). They'reON DELETE RESTRICT ON UPDATE CASCADE, like the join tables. RESTRICT is whatLanguage\Gateway::canDeleteLanguage()already checks before a language is deleted, so the API behaves the same, the database just enforces it now.deleteLanguage()removes a language's search index rows before the language itself, so those don't block it. The field definition translations' foreign key wasON DELETE CASCADE, it's re-created as RESTRICT under a new name,ibexa_content_type_field_definition_ml_language_fk.Each foreign key has its own migration, like
AddContentFieldLanguageForeignKeyMigration, 13 of them on top ofAbstractAddLanguageForeignKeyMigration, running after the bitmask migrations. MySQL and MariaDB can't roll back schema changes, so with one migration per key a failure leaves the keys added before it recorded as executed, andibexa:doctrine:migrations:statusshows where it stopped. Each one refuses while its column holds an id that isn't a language, naming the column and the row count, and skips itself when its key is already there. On MySQL it's a singleALTER TABLE, adding the key together with the index MySQL needs on a foreign key column (also inschema.yaml, so both install paths match). SQLite can't add a foreign key to an existing table, so there it rebuilds the table with its original DDL, and refuses to run whilePRAGMA foreign_keysis on, as dropping a table would then delete the rows referencing it.Tests needed the languages in place before anything referencing them:
schema.yamllists the language table first (the test schema is dropped in reverse), the clean data inserts it first, andFixtureImporterinserts it first and truncates it last. When a fixture brings its own languages,FixtureImporternow sets aside the rows of every table referencing a language, not only the translation tables, and puts back the ones whose languages still exist. Fixtures using ids of languages they didn't have got those languages. The legacy search fixture's index rows had no language at all and now have eng-US's id, andFixtureImporterturns a search index row's legacylanguage_maskintolanguage_idandis_main_and_always_available, since test-core'stest_data.yamlstill has it.For QA:
By hand: upgrade a 5.0 database with
ibexa:doctrine:migrations:migrate, then check that a language used by content or a content type can't be deleted, and that an unused one can.Documentation:
For the 6.0 upgrade guide: the
Add*LanguageForeignKeyMigrationmigrations stop the upgrade when a language id column refers to a language that doesn't exist anymore, naming the column. Those rows have to be deleted, or pointed at an existing language, before migrating again.