Skip to content

IBX-13142: Added foreign keys from the language id columns to the language table - #964

Open
Steveb-p wants to merge 11 commits into
feature/language-bitmask-6-remove-6.0from
feature/language-foreign-keys-6.0
Open

Steveb-p wants to merge 11 commits into
feature/language-bitmask-6-remove-6.0from
feature/language-foreign-keys-6.0

Conversation

@Steveb-p

@Steveb-p Steveb-p commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor
🎫 Issue IBX-13142

Note

Stacked on #946, the last part of the language bitmask removal. #965 goes on top of it.

Related PRs:

Description:

Only #941's join tables and the field definition translations had a foreign key to ibexa_content_language. Now every core column holding a language id does: the initial language of content, versions and content types, the language of fields, content names, content type names and object state translations, the default language of object states and groups, and the legacy search index (ibexa_search_object_word_link). They're ON DELETE RESTRICT ON UPDATE CASCADE, like the join tables. RESTRICT is what Language\Gateway::canDeleteLanguage() already checks before a language is deleted, so the API behaves the same, the database just enforces it now. deleteLanguage() removes a language's search index rows before the language itself, so those don't block it. The field definition translations' foreign key was ON DELETE CASCADE, it's re-created as RESTRICT under a new name, ibexa_content_type_field_definition_ml_language_fk.

Each foreign key has its own migration, like AddContentFieldLanguageForeignKeyMigration, 13 of them on top of AbstractAddLanguageForeignKeyMigration, running after the bitmask migrations. MySQL and MariaDB can't roll back schema changes, so with one migration per key a failure leaves the keys added before it recorded as executed, and ibexa:doctrine:migrations:status shows where it stopped. Each one refuses while its column holds an id that isn't a language, naming the column and the row count, and skips itself when its key is already there. On MySQL it's a single ALTER TABLE, adding the key together with the index MySQL needs on a foreign key column (also in schema.yaml, so both install paths match). SQLite can't add a foreign key to an existing table, so there it rebuilds the table with its original DDL, and refuses to run while PRAGMA foreign_keys is on, as dropping a table would then delete the rows referencing it.

Tests needed the languages in place before anything referencing them: schema.yaml lists the language table first (the test schema is dropped in reverse), the clean data inserts it first, and FixtureImporter inserts it first and truncates it last. When a fixture brings its own languages, FixtureImporter now sets aside the rows of every table referencing a language, not only the translation tables, and puts back the ones whose languages still exist. Fixtures using ids of languages they didn't have got those languages. The legacy search fixture's index rows had no language at all and now have eng-US's id, and FixtureImporter turns a search index row's legacy language_mask into language_id and is_main_and_always_available, since test-core's test_data.yaml still has it.

For QA:

vendor/bin/phpunit -c phpunit.xml tests/bundle/RepositoryInstaller/Migration/LanguageBitmaskUpgradeSequenceTest.php

By hand: upgrade a 5.0 database with ibexa:doctrine:migrations:migrate, then check that a language used by content or a content type can't be deleted, and that an unused one can.

Documentation:

For the 6.0 upgrade guide: the Add*LanguageForeignKeyMigration migrations stop the upgrade when a language id column refers to a language that doesn't exist anymore, naming the column. Those rows have to be deleted, or pointed at an existing language, before migrating again.

@Steveb-p
Steveb-p force-pushed the feature/language-foreign-keys-6.0 branch from 28485ee to 565498f Compare November 17, 2026 11:19
@Steveb-p
Steveb-p force-pushed the feature/language-bitmask-6-remove-6.0 branch from 3cbeb68 to 153157f Compare November 17, 2026 11:19
@Steveb-p
Steveb-p force-pushed the feature/language-foreign-keys-6.0 branch from 565498f to 1bef7c1 Compare November 17, 2026 11:50
@Steveb-p
Steveb-p marked this pull request as ready for review November 17, 2026 14:15
@Steveb-p
Steveb-p force-pushed the feature/language-bitmask-6-remove-6.0 branch from 8bb44d5 to 42998d1 Compare November 18, 2026 11:50
@Steveb-p
Steveb-p force-pushed the feature/language-foreign-keys-6.0 branch from 32391a2 to 31972d9 Compare November 18, 2026 11:50
@Steveb-p
Steveb-p force-pushed the feature/language-bitmask-6-remove-6.0 branch from 42998d1 to 8409439 Compare November 18, 2026 12:13
@Steveb-p
Steveb-p force-pushed the feature/language-foreign-keys-6.0 branch from 31972d9 to b082d9e Compare November 18, 2026 12:13
@Steveb-p
Steveb-p force-pushed the feature/language-bitmask-6-remove-6.0 branch from 8409439 to f0f52b3 Compare November 18, 2026 13:06
@Steveb-p
Steveb-p force-pushed the feature/language-foreign-keys-6.0 branch from d502579 to 9f380d6 Compare November 18, 2026 13:06
@@ -0,0 +1,4 @@
ALTER TABLE ibexa_content_field

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With foreign_key_checks on, MySQL adds a foreign key with ALGORITHM=COPY only: 13 table copies with writes blocked, ibexa_content_field and ibexa_search_object_word_link included. How is it going to behave on MySQL / MariaDB / PostgreSQL with a bigger dataset? CI executes the SQLite files only.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've measured it locally on 300k content items, with 3M fields (~1.4–1.9 GB) and 10M word links (~1.8–2 GB), default 128 MB buffers. Total time of all 13 migrations:

As is foreign_key_checks = 0
MySQL 8.0 353 s 15 s
MariaDB 10.11 135 s 18 s
PostgreSQL 16 6 s –

Almost all of the MySQL/MariaDB time is the table copy: 304 s (MySQL) and 89 s (MariaDB) for ibexa_search_object_word_link alone, and ~25 s for ibexa_content_field. With the checks on, both reject ALGORITHM=INPLACE ("Adding foreign keys needs foreign_key_checks=OFF"). With them off, both accept ALGORITHM=INPLACE, LOCK=NONE, so the index build that's left doesn't block writes either. PostgreSQL validates each key in a single scan (0.5 s at most here), and NOT VALID + VALIDATE CONSTRAINT didn't change anything, so I'd leave it as it is.

The time grows with the table size, so a site with 10x the word links would spend close to an hour copying tables as it is now. I'd switch foreign_key_checks off around the ALTERs on MySQL/MariaDB, the same way #964 now switches PRAGMA foreign_keys off on SQLite. MySQL doesn't validate the existing rows then, but the migration checks for missing language ids right before, so only rows written during the upgrade itself could slip through.

Should we go with it? @alongosz, what's your take?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The time grows with the table size, so a site with 10x the word links would spend close to an hour copying tables as it is now. I'd switch foreign_key_checks off around the ALTERs on MySQL/MariaDB (...)

Should we go with it? @alongosz, what's your take?

Makes sense

@@ -0,0 +1,3 @@
ALTER TABLE ibexa_content_type_field_definition_ml

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No index in this variant? If the old key is missing, the ibexa_content_type_field_definition_ml_lang_fk index from schema.yaml may be missing as well. Same for the MySQL one.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in d46cb6a: when no index on the table starts with language_id, the migration creates ibexa_content_type_field_definition_ml_lang_fk before adding the key, in both variants. It looks for any such index rather than that name, since sites upgraded with the installer's 5.0 script still have it as ezcontentclass_attribute_ml_lang_fk.


if ($this->isSqlite()) {
$this->abortIf(
(bool)$this->connection->fetchOne('PRAGMA foreign_keys'),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question as on #801: is the SQLite rebuild needed? 13 table definitions to keep in sync with schema.yaml, plus this abort, and #965 already renumbers without the keys there.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebuilding the table is how SQLite itself recommends making changes ALTER TABLE can't: https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes. I've dropped the abort in 9997c72, too: on SQLite the migrations now run outside a transaction, so they can switch PRAGMA foreign_keys off for the rebuild and back on afterwards, as that procedure does. The definitions don't have to follow schema.yaml later on either. Each one is the table as of this migration, and later changes come as new migrations. We only use SQLite for testing, so it's best-effort anyway, and I'd keep it.

name: { type: string, nullable: false, length: 45, options: { default: '' } }
foreignKeys:
ibexa_object_state_group_language_language_fk: { fields: [language_id], foreignTable: ibexa_content_language, foreignFields: [id], options: { onDelete: RESTRICT, onUpdate: CASCADE } }
ibexa_object_state_group_language_real_language_fk: { fields: [real_language_id], foreignTable: ibexa_content_language, foreignFields: [id], options: { onDelete: RESTRICT, onUpdate: CASCADE } }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

canDeleteLanguage() checks only language_id for this table ($columns[0]). Add real_language_id to MULTILINGUAL_TABLES_COLUMNS?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 989b490. canDeleteLanguage() also checked only the first column listed for each table, so it goes through all of them now.

data_json: { type: text, nullable: true, length: 65535 }
foreignKeys:
ibexa_content_type_field_definition_ml_lang_fk: { fields: [language_id], foreignTable: ibexa_content_language, foreignFields: [id], options: { onDelete: CASCADE, onUpdate: CASCADE } }
ibexa_content_type_field_definition_ml_language_fk: { fields: [language_id], foreignTable: ibexa_content_language, foreignFields: [id], options: { onDelete: RESTRICT, onUpdate: CASCADE } }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the rename? The 5.0 upgrade script just gave it the _ml_lang_fk name, and the index above keeps it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not only the name: the key goes from ON DELETE CASCADE to RESTRICT, as the other 12 do. Neither MySQL nor PostgreSQL can change that in place, so it's dropped and re-added either way, and the new name comes for free with that. It lines it up with the other <table>_language_fk keys, and the "already added" check works the same for all 13. RESTRICT matches what core already does, too: canDeleteLanguage() refuses to delete a language this table still uses, so the CASCADE could only kick in on a direct SQL delete.

status: { type: integer, nullable: true, options: { default: '0' } }
is_hidden: { type: boolean, nullable: false, options: { default: '0' } }
foreignKeys:
ibexa_content_initial_language_fk: { fields: [initial_language_id], foreignTable: ibexa_content_language, foreignFields: [id], options: { onDelete: RESTRICT, onUpdate: CASCADE } }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default 0 can never pass this key, the fixtures here needed languages and initial_language_id added. Are consumers (corporate-account, mcp, measurement, data-intelligence-layer) ready?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They should be. corporate-account's and mcp's fixtures set initial_language_id (2, and 8 in mcp) on every content and version row and go through FixtureImporter, and measurement doesn't seed any content. data-intelligence-layer sets initial_language_id on its content inserts since ibexa/data-intelligence-layer#63, though its CI ran against #946, not this PR yet.

*
* @return array<string, mixed>
*/
private function splitLegacySearchIndexLanguageMask(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

full_dump.php got language_id on all its rows, test_data.yaml keeps language_mask: 3 and goes through this. Why not fix that fixture as well?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

full_dump.php had nothing to convert: its search index rows are older than both language_mask and language_id, so they'd all have landed on language 0. test_data.yaml has the mask, and FixtureImporter splits it like the other masks in that file. The kernel suite reads test-core's own test_data.yaml anyway, which still has the mask, so the conversion has to stay either way.

@Steveb-p
Steveb-p force-pushed the feature/language-bitmask-6-remove-6.0 branch from f0f52b3 to bcd1688 Compare November 18, 2026 15:35
@Steveb-p
Steveb-p force-pushed the feature/language-foreign-keys-6.0 branch from 9f380d6 to 86e1c98 Compare November 18, 2026 15:35
…guage table

The initial language of content, versions and content types, the language of fields,
content names, content type names and object state translations, and the default
language of object states and groups now reference ibexa_content_language, ON DELETE
RESTRICT ON UPDATE CASCADE like the translation tables. The field definition
translations' foreign key, ON DELETE CASCADE until now, follows suit under a new name.

AddLanguageForeignKeysMigration adds them after the bitmask migrations, refusing while
any row holds a language id without a language. SQLite rebuilds the tables.

schema.yaml lists the language table first and the clean data inserts it first;
FixtureImporter inserts it first and detaches every row referencing it when a fixture
replaces the languages; fixtures and tests use language ids that exist.
…gn key

MySQL and MariaDB can't roll back schema changes, so a failure halfway
through the single migration left some of the foreign keys added and
others not. Each foreign key now has its own migration, which adds it
together with its index in one statement and skips itself when the key
is already there, so the migrations' status shows which ones are in.
…key rebuilds

The migrations run outside a transaction on SQLite, so PRAGMA foreign_keys can be switched off around the table rebuild, and back on afterwards, instead of aborting when the connection enforces foreign keys.
@Steveb-p
Steveb-p force-pushed the feature/language-foreign-keys-6.0 branch from 9997c72 to b6a64be Compare November 21, 2026 16:16
@Steveb-p
Steveb-p force-pushed the feature/language-bitmask-6-remove-6.0 branch from bcd1688 to 7f0bb8d Compare November 21, 2026 16:16
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants