Repository navigation
Conversation
cf5c30f to
4f76bce
Compare
4f76bce to
8d2f0bc
Compare
konradoboza
left a comment
There was a problem hiding this comment.
IBX-11939 is the automatic Doctrine SQL migrations ticket, a dedicated one for the bitmask removal please. The migration path for existing installations is my main concern, remarks inline:
| */ | ||
| private function resolveLanguageTableName(AbstractSchemaManager $schemaManager): string | ||
| { | ||
| return $schemaManager->tablesExist(['ibexa_content_language']) |
There was a problem hiding this comment.
Claude finding
tablesExist() goes through the connection's schema assets filter, and ManagedTablesSchemaAssetFilter is tagged on every repository connection (one ibexa_<conn> EM per connection), so ibexa_content_language is always reported as missing and this always falls back to ibexa_language. After #801 that's right only by luck - a manual repair after a failed backfill, before the rename migration runs, still hits a table that doesn't exist yet. FixtureImporter in this PR already does this check through SchemaAssetsFilterBypassInterface - shouldn't both commands do the same? Same in BackfillLanguageTranslationsCommand::resolveLanguageTableName()
| * | ||
| * @param array<array<string, scalar>> $rows | ||
| */ | ||
| protected function getLanguageGatewayStub( |
There was a problem hiding this comment.
This takes version languages from field rows, not from the version mask the fixture had (content_version_language_mask => 3, so eng-US only, while one field row is eng-GB). Because of that the expected content gained eng-GB in languageCodes plus eng-GB virtual fields, and the converter counts went from count($rowsFixture) - 1 to null. So now the test checks what the stub returns, not what a backfill from the mask would give. Can we feed it languages decoded from the fixture mask and keep the old expectations and counts?
There was a problem hiding this comment.
Done in 915f860: the tests get the languages the old masks encoded ([676 => [2]]), so the expectations and the converter counts are back to what they were on 6.0.
| } | ||
| } | ||
|
|
||
| $gateway = $this->createMock(LanguageGateway::class); |
There was a problem hiding this comment.
| $gateway = $this->createMock(LanguageGateway::class); | |
| $gateway = $this->createStub(LanguageGateway::class); |
it only feeds data, no expectations on it
There was a problem hiding this comment.
The stub is gone since f134336, the Mapper takes the version languages as an argument now.
| * Purely additive: tables are created empty and stay empty until | ||
| * "ibexa:languages:backfill-translations" populates them, and nothing reads them yet - the mask | ||
| * columns remain authoritative until later steps switch read paths over and eventually drop them. |
There was a problem hiding this comment.
This is outdated - BackfillLanguageTranslationsMigration fills these tables, and Mapper, the content/location gateways and Filter read from them in this PR. The same goes for the DropLanguageBitmaskColumnsMigration mentions in the backfill migration and both commands (VerifyLanguageTranslationsCommand even has a use for it), and that class isn't here. Also the "stray pre-migration always available bit" comments in Filter's DoctrineGateway - updateContentNameAlwaysAvailableFlag() still writes that bit to content_name.language_id, so it's not a leftover
|
Created IBX-13142 for the bitmask removal, and moved the PRs over to it. |
konradoboza
left a comment
There was a problem hiding this comment.
+1 as long as Dawid's remarks are taken into account.
| } | ||
|
|
||
| if ($missing > 0) { | ||
| $inserted = $this->connection->executeStatement($queries['fixMissing']); |
There was a problem hiding this comment.
The backfill runs in chunks of 5000 ids because these tables can be big, but --fix does the same insert as one statement over the whole table, and so do the orphan DELETE and the always_available UPDATE below. The description points sites at verify-translations --fix for URL aliases created before #943, so it will run on those same tables. Is a single statement over the whole table intended here?
There was a problem hiding this comment.
Not intended, no. Changed in f737487: --fix now runs the insert, the orphan DELETE and the always_available UPDATE in batches of 5000 ids, like the backfill.
| // truncate-then-insert loop never clears them directly; do it explicitly rather than rely | ||
| // on FK cascade behavior varying by platform/driver. | ||
| if (!empty($nonEmptyTablesData['ibexa_content'])) { | ||
| $this->connection->executeStatement('DELETE FROM ibexa_content_translation'); |
There was a problem hiding this comment.
For an AppendOnlyFixture we skip the truncate, but this DELETE still clears the whole join table and the loop below re-inserts rows only for the content in the fixture. Content that was already there (for example from ImportDataMigration) ends up with no languages, and Mapper reads them from here now. Is that intended?
There was a problem hiding this comment.
Not intended. Fixed in 6fd903e: an AppendOnlyFixture doesn't clear the translation tables anymore, so content that's already there keeps its languages.
| ): array { | ||
| return match ($table) { | ||
| self::TABLE_CONTENT => [ | ||
| 'missing' => "SELECT COUNT(*) FROM ibexa_content c |
There was a problem hiding this comment.
could we use heredoc for these?
| */ | ||
| private function reattachLanguageTranslations(array $detachedTranslations): void | ||
| { | ||
| if (empty($detachedTranslations)) { |
There was a problem hiding this comment.
could we make stricter checks here?
There was a problem hiding this comment.
Done in 7838cb0: === [] here, and isset() on $nonEmptyTablesData, since it only ever holds non-empty tables.
| $this->connection->executeStatement('DELETE FROM ibexa_url_alias_ml_translation'); | ||
| } | ||
|
|
||
| if (!empty($nonEmptyTablesData['ibexa_content'])) { |
There was a problem hiding this comment.
same as above and in other occurences of empty
There was a problem hiding this comment.
Same commit, 7838cb0, covers every other empty() in FixtureImporter.
| * | ||
| * @return array<int, int[]> Version id => language ids | ||
| */ | ||
| abstract public function loadVersionTranslations(array $versionIds): array; |
There was a problem hiding this comment.
should it be cached additionally? afaics it's used often on content load
There was a problem hiding this comment.
I'd leave it uncached here. The persistence cache already caches loaded content and version infos together with their languages, so this only runs on a cache miss, and then once per list of versions rather than per version.
There was a problem hiding this comment.
Claude:
General remark: the backfill and verification logic is written as verbatim SQL in 3 places. The
INSERT … JOIN … ON (language_mask & l.id) = l.idstatements inBackfillLanguageTranslationsCommand(L101-123) are copies of the ones inBackfillLanguageTranslationsMigration(L62-84), andVerifyLanguageTranslationsCommandhas a 4th copy in itsfixMissingqueries, next to the samealways_availableupdate from bit 0.resolveLanguageTableName()is duplicated between both commands too.
This finding came from the fact that I don't like in general SQL-s verbatim in Command implementations. All these migration SQL-s could have a dedicated gateway, especially as it seems some methods can be reused.
(don't worry - it definitely doesn't belong in Content Gateway ;))
f4ecdbd to
6957c1f
Compare
…in tables Adds ibexa_content_translation and ibexa_content_version_translation (and ibexa_url_alias_ml_translation, which the same migration creates) and an always_available column on ibexa_content and ibexa_content_version, backfills them from the language masks, and switches the content and location gateways, the content mapper and the Filter to them. The content gateway keeps writing the masks and the always-available bit of the field and name language ids, which the remaining readers still use until the bitmask is removed. Adds the ibexa:languages:backfill-translations and ibexa:languages:verify-translations commands, and the new columns and rows to cleandata.sql. FixtureImporter fills the join tables for fixtures that still carry masks, and creates its SchemaAssetsFilterBypass itself when it isn't given one.
…illLanguageTranslationsMigration
…assed them to the Mapper
… in the language translation commands
…spectTableByUnquotedName()
…join table code IBX-13070 removed SchemaAssetsFilterBypass together with the filter it lifted, so the language translation commands and FixtureImporter introspect the connection directly again.
6957c1f to
b5273d3
Compare
…ppend-only fixture
…th the queries as heredocs
…clared the commands' exceptions
…nguage backfill per table
…d declared its exceptions
| $gatewayMock->expects($this->once()) | ||
| ->method('loadVersionTranslations') | ||
| ->with([22, 31]) | ||
| ->willReturn($versionLanguageIds); |
There was a problem hiding this comment.
[nit] loadVersionTranslations() only feeds data, so stub it instead of expects($this->once()) - the mapper callback already asserts what got passed.
|



Note
Part 1 of 6 of the language bitmask removal. Each part is green on its own; the bitmask keeps being written until #946 removes it.
Related PRs:
The stack that replaces the language bitmask, bottom to top. Each part is green and can be merged on its own, in this order:
MaskGeneratoribexa/installer#246 goes first, as the Experience content it provisions has to match each step, and ibexa/installer#250 drops its schema checks after #946. ibexa/data-intelligence-layer#63 goes with #946.
Description:
Content and its versions keep their languages in
ibexa_content_translationandibexa_content_version_translation, and "always available" gets its ownalways_availablecolumn instead of living in bit 0 of the mask. The content and location gateways and the Filter read them from there, and the content handler loads each version's languages and passes them toContent\Mapper. The content gateway still writeslanguage_maskand the always-available bit of the field and name language ids, because the legacy search engine, URL aliases andcanDeleteLanguage()keep reading them until the next PRs move them over.AddContentAlwaysAvailableColumnsMigrationadds the columns andAddLanguageTranslationTablesMigrationcreates the join tables.BackfillLanguageTranslationsMigrationfills both from the masks, in chunks of 5000 ids and outside a transaction, so rerunning it after a failure finishes the job. It also coversibexa_url_alias_ml_translation, which #943 starts using.ibexa:languages:backfill-translationsandibexa:languages:verify-translationsdo the same by hand and check the result,always_availableincluded. A site that runs this PR's migrations and then creates URL aliases before #943 lands won't have those inibexa_url_alias_ml_translation;ibexa:languages:verify-translations --fixadds them.cleandata.sqlgets the new columns and rows for the legacy install path.FixtureImporterfills the join tables for fixtures that still carry masks, as most packages' test data does, and creates itsSchemaAssetsFilterBypassitself when it isn't given one, so packages using test-core's service definition keep working.Content\GatewaygainsloadContentTranslations()andloadVersionTranslations(), so a class extending it directly breaks.Content\Mapper::extractContentFromRows()andextractVersionInfoListFromRows()take the version languages as a new third argument.