Skip to content

IBX-13142: Moved content and version languages from the bitmask to join tables - #941

Open
Steveb-p wants to merge 24 commits into
6.0from
feature/language-bitmask-1-content-6.0
Open

Steveb-p wants to merge 24 commits into
6.0from
feature/language-bitmask-1-content-6.0

Conversation

@Steveb-p

@Steveb-p Steveb-p commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor
🎫 Issue IBX-13142

Note

Part 1 of 6 of the language bitmask removal. Each part is green on its own; the bitmask keeps being written until #946 removes it.

Related PRs:

The stack that replaces the language bitmask, bottom to top. Each part is green and can be merged on its own, in this order:

  1. this PR: content and version languages in join tables
  2. IBX-13142: Moved the legacy search engine off the language bitmask #942: legacy search engine off the bitmask
  3. IBX-13142: Moved URL aliases off the language bitmask #943: URL aliases off the bitmask
  4. IBX-13142: Moved content types off the language bitmask #944: content types off the bitmask
  5. IBX-13142: Moved object states off MaskGenerator #945: object states off MaskGenerator
  6. IBX-13142: Removed the language bitmask #946: removed the bitmask

ibexa/installer#246 goes first, as the Experience content it provisions has to match each step, and ibexa/installer#250 drops its schema checks after #946. ibexa/data-intelligence-layer#63 goes with #946.

Description:

Content and its versions keep their languages in ibexa_content_translation and ibexa_content_version_translation, and "always available" gets its own always_available column instead of living in bit 0 of the mask. The content and location gateways and the Filter read them from there, and the content handler loads each version's languages and passes them to Content\Mapper. The content gateway still writes language_mask and the always-available bit of the field and name language ids, because the legacy search engine, URL aliases and canDeleteLanguage() keep reading them until the next PRs move them over.

AddContentAlwaysAvailableColumnsMigration adds the columns and AddLanguageTranslationTablesMigration creates the join tables. BackfillLanguageTranslationsMigration fills both from the masks, in chunks of 5000 ids and outside a transaction, so rerunning it after a failure finishes the job. It also covers ibexa_url_alias_ml_translation, which #943 starts using. ibexa:languages:backfill-translations and ibexa:languages:verify-translations do the same by hand and check the result, always_available included. A site that runs this PR's migrations and then creates URL aliases before #943 lands won't have those in ibexa_url_alias_ml_translation; ibexa:languages:verify-translations --fix adds them.

cleandata.sql gets the new columns and rows for the legacy install path. FixtureImporter fills the join tables for fixtures that still carry masks, as most packages' test data does, and creates its SchemaAssetsFilterBypass itself when it isn't given one, so packages using test-core's service definition keep working.

Content\Gateway gains loadContentTranslations() and loadVersionTranslations(), so a class extending it directly breaks. Content\Mapper::extractContentFromRows() and extractVersionInfoListFromRows() take the version languages as a new third argument.

@konradoboza konradoboza left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IBX-11939 is the automatic Doctrine SQL migrations ticket, a dedicated one for the bitmask removal please. The migration path for existing installations is my main concern, remarks inline:

*/
private function resolveLanguageTableName(AbstractSchemaManager $schemaManager): string
{
return $schemaManager->tablesExist(['ibexa_content_language'])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude finding

tablesExist() goes through the connection's schema assets filter, and ManagedTablesSchemaAssetFilter is tagged on every repository connection (one ibexa_<conn> EM per connection), so ibexa_content_language is always reported as missing and this always falls back to ibexa_language. After #801 that's right only by luck - a manual repair after a failed backfill, before the rename migration runs, still hits a table that doesn't exist yet. FixtureImporter in this PR already does this check through SchemaAssetsFilterBypassInterface - shouldn't both commands do the same? Same in BackfillLanguageTranslationsCommand::resolveLanguageTableName()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in faf93ab: both commands check for the mask column through SchemaAssetsFilterBypassInterface now, with a test that hides every table behind a filter. The language table lookup that commit added is gone again since f4ecdbd, as the table keeps its name.

*
* @param array<array<string, scalar>> $rows
*/
protected function getLanguageGatewayStub(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This takes version languages from field rows, not from the version mask the fixture had (content_version_language_mask => 3, so eng-US only, while one field row is eng-GB). Because of that the expected content gained eng-GB in languageCodes plus eng-GB virtual fields, and the converter counts went from count($rowsFixture) - 1 to null. So now the test checks what the stub returns, not what a backfill from the mask would give. Can we feed it languages decoded from the fixture mask and keep the old expectations and counts?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 915f860: the tests get the languages the old masks encoded ([676 => [2]]), so the expectations and the converter counts are back to what they were on 6.0.

}
}

$gateway = $this->createMock(LanguageGateway::class);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
$gateway = $this->createMock(LanguageGateway::class);
$gateway = $this->createStub(LanguageGateway::class);

it only feeds data, no expectations on it

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stub is gone since f134336, the Mapper takes the version languages as an argument now.

Comment on lines +25 to +27
* Purely additive: tables are created empty and stay empty until
* "ibexa:languages:backfill-translations" populates them, and nothing reads them yet - the mask
* columns remain authoritative until later steps switch read paths over and eventually drop them.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is outdated - BackfillLanguageTranslationsMigration fills these tables, and Mapper, the content/location gateways and Filter read from them in this PR. The same goes for the DropLanguageBitmaskColumnsMigration mentions in the backfill migration and both commands (VerifyLanguageTranslationsCommand even has a use for it), and that class isn't here. Also the "stray pre-migration always available bit" comments in Filter's DoctrineGateway - updateContentNameAlwaysAvailableFlag() still writes that bit to content_name.language_id, so it's not a leftover

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trimmed in 5e5cac7. The Filter comments were wrong as you say, the bit is still written until #946, which is also where the + 1 has to go.

@Steveb-p Steveb-p changed the title IBX-11939: Moved content and version languages from the bitmask to join tables IBX-13142: Moved content and version languages from the bitmask to join tables Nov 13, 2026
@Steveb-p

Copy link
Copy Markdown
Contributor Author

Created IBX-13142 for the bitmask removal, and moved the PRs over to it.

@konradoboza konradoboza left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 as long as Dawid's remarks are taken into account.

}

if ($missing > 0) {
$inserted = $this->connection->executeStatement($queries['fixMissing']);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The backfill runs in chunks of 5000 ids because these tables can be big, but --fix does the same insert as one statement over the whole table, and so do the orphan DELETE and the always_available UPDATE below. The description points sites at verify-translations --fix for URL aliases created before #943, so it will run on those same tables. Is a single statement over the whole table intended here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not intended, no. Changed in f737487: --fix now runs the insert, the orphan DELETE and the always_available UPDATE in batches of 5000 ids, like the backfill.

// truncate-then-insert loop never clears them directly; do it explicitly rather than rely
// on FK cascade behavior varying by platform/driver.
if (!empty($nonEmptyTablesData['ibexa_content'])) {
$this->connection->executeStatement('DELETE FROM ibexa_content_translation');

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For an AppendOnlyFixture we skip the truncate, but this DELETE still clears the whole join table and the loop below re-inserts rows only for the content in the fixture. Content that was already there (for example from ImportDataMigration) ends up with no languages, and Mapper reads them from here now. Is that intended?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not intended. Fixed in 6fd903e: an AppendOnlyFixture doesn't clear the translation tables anymore, so content that's already there keeps its languages.

): array {
return match ($table) {
self::TABLE_CONTENT => [
'missing' => "SELECT COUNT(*) FROM ibexa_content c

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we use heredoc for these?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f737487.

*/
private function reattachLanguageTranslations(array $detachedTranslations): void
{
if (empty($detachedTranslations)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we make stricter checks here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 7838cb0: === [] here, and isset() on $nonEmptyTablesData, since it only ever holds non-empty tables.

$this->connection->executeStatement('DELETE FROM ibexa_url_alias_ml_translation');
}

if (!empty($nonEmptyTablesData['ibexa_content'])) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above and in other occurences of empty

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same commit, 7838cb0, covers every other empty() in FixtureImporter.

*
* @return array<int, int[]> Version id => language ids
*/
abstract public function loadVersionTranslations(array $versionIds): array;

@barw4 barw4 Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should it be cached additionally? afaics it's used often on content load

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd leave it uncached here. The persistence cache already caches loaded content and version infos together with their languages, so this only runs on a cache miss, and then once per list of versions rather than per version.

@alongosz alongosz left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude:

General remark: the backfill and verification logic is written as verbatim SQL in 3 places. The INSERT … JOIN … ON (language_mask & l.id) = l.id statements in BackfillLanguageTranslationsCommand (L101-123) are copies of the ones in BackfillLanguageTranslationsMigration (L62-84), and VerifyLanguageTranslationsCommand has a 4th copy in its fixMissing queries, next to the same always_available update from bit 0. resolveLanguageTableName() is duplicated between both commands too.

This finding came from the fact that I don't like in general SQL-s verbatim in Command implementations. All these migration SQL-s could have a dedicated gateway, especially as it seems some methods can be reused.

(don't worry - it definitely doesn't belong in Content Gateway ;))

…in tables

Adds ibexa_content_translation and ibexa_content_version_translation
(and ibexa_url_alias_ml_translation, which the same migration creates)
and an always_available column on ibexa_content and
ibexa_content_version, backfills them from the language masks, and
switches the content and location gateways, the content mapper and the
Filter to them. The content gateway keeps writing the masks and the
always-available bit of the field and name language ids, which the
remaining readers still use until the bitmask is removed.

Adds the ibexa:languages:backfill-translations and
ibexa:languages:verify-translations commands, and the new columns and
rows to cleandata.sql. FixtureImporter fills the join tables for
fixtures that still carry masks, and creates its SchemaAssetsFilterBypass
itself when it isn't given one.
@Steveb-p
Steveb-p force-pushed the feature/language-bitmask-1-content-6.0 branch from 6957c1f to b5273d3 Compare November 18, 2026 11:49
Comment on lines +545 to +548
$gatewayMock->expects($this->once())
->method('loadVersionTranslations')
->with([22, 31])
->willReturn($versionLanguageIds);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] loadVersionTranslations() only feeds data, so stub it instead of expects($this->once()) - the mapper callback already asserts what got passed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stubbed in adadf24.

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants