Feature #12584
closedFeature #12440: Typo3 V10 upgrade (durchfuehren und testen)
T3 v10 migration script: replace alias-patterns (v11)
100%
Description
ALTE VERSION von Spezifikation! neue Version im Kommentar¶
Functional Spec:¶
- If in any Report or Form there is an occurrence of an "id=alias" pattern (see bellow) and the T3 version is higher than 9, then QFQ shows an exception with the following content:
- link to automatically apply all suggested replacements
- link to reload and skip this check and exception forever
- Detailed list of each occurrence together with the suggested replacement
- If no suggested id was found for an alias then this is marked prominently in the error message together with the location of the occurrence
- patterns where <alias> will be replaced by id
href="?id=<alias>&
href='?id=<alias>&
href="?id=<alias>"
href='?id=<alias>'
- patterns where
{{pageAlias:T}}
will be replaced by{{pageId:T}}
href="?id={{pageAlias:T}}
href='?id={{pageAlias:T}}
- If one of the following are true then QFQ never shows this exception again:
- The search found no occurrence of any of the above patterns
- "skip check forever" link was clicked
- Question A : should all occurrences of
{{pageAlias:T}}
be replaced by{{pageId:T}}
everywhere even when there is no "href"?- The page alias will not have any meaning or won't even exist for Typo3 v10 upwards
- alternatively we could introduce a new variable
{{pagePath:T}}
. But I don't know how easy it will be to compute that path reliably and I would rather not ask Typo3 to do it...
Technical Spec:¶
- the following columns are checked for occurrences of `id=alias`
- tt_content.bodytext
- Form.title
- Form.parameter
- Form.parameterLanguageA/B/C/D
- FormElement.label
- FormElement.note
- FormElement.value
- FormElement.sql1
- FormElement.parameter
- FormElement.parameterLanguageA/B/C/D
- The page id for patterns containing
<alias>
will be found the following way- search <alias> in column
pages.slug
- when migrating from alias to slug underscore "_" was replaced by minus "-" by Typo3 automatically, so search again with "_" replaced in <alias>
- if not found, search in column
pages.zzz_deleted_alias
if that column exists - if not found mark <alias> prominently as not found in error message.
- search <alias> in column
- If the migration was performed or skipped then the flag
skipAliasMigration
is added to the comment of the Form table- The current comment looks like this:
Version=21.5.0a&functionHash=dec0567ea030ed61fe7cee8dbba2a935&functionVersion=20.11.0
- The current comment looks like this:
- Variants to check if Typo3 version is higher than 9
- The Typo3 version might be found in a Global variable?
- if not, the symlink in the root folder could be parsed:
typo3_src -> /var/www/typo3_src-8.7.32/
- or the hacky way it is done in the password hashing function by checking if a certain Typo3 class exists
Updated by Marc Egger almost 3 years ago
- Subject changed from T3 v10 migration script: replace all id=alias with id to T3 v10 migration script: replace id=alias patterns with id
Updated by Marc Egger almost 3 years ago
- Status changed from New to Feedback
- Assignee changed from Marc Egger to Carsten Rose
Bitte um feadback und deine Meinung zu "Question A"
Updated by Marc Egger almost 3 years ago
- Status changed from Feedback to In Progress
- Assignee changed from Carsten Rose to Marc Egger
Updated by Marc Egger almost 3 years ago
- Description updated (diff)
neue Version nach Besprechung Carsten (danach nicht mehr aktuallisiert, siehe Release.rst fuer aktuelle Infos)¶
General¶
- alias wird bei der Migration komplett mit slug ersetzt
- nach der Migration wird davon ausgegangen, dass nur noch slug verwendet wird
Functional Spec:¶
- If in any Report or Form there is an occurrence of an "alias-pattern" (see bellow) and the T3 version is higher than 9, then QFQ shows an exception with the following content:
- link to automatically apply all suggested replacements
- link to reload and skip this check and exception forever
- Detailed list of each occurrence together with the suggested replacement
- If no suggestion was found for a pattern then this is marked prominently in the error message together with the location of the occurrence
- patterns where
<alias/id>
will be replaced by<slug>
p:<alias/id>&
p:<alias/id>|
"p:<alias/id>"
'p:<alias/id>'
|p:<alias/id>'
|p:id=<alias/id>'
- patterns where
{{pageAlias:T}}
will be replaced by{{pageSlug:T}}
"p:{{pageAlias:T}}
'p:{{pageAlias:T}}
|p:{{pageAlias:T}}
'p:id={{pageAlias:T}}
- patterns where
<alias>
will be replaced by{{baseUrlLang:Y}}/<slug>
href="[index.php]?id=<alias/id>&
href='[index.php]?id=<alias/id>&
href="[index.php]?id=<alias/id>"
href='[index.php]?id=<alias/id>'
- patterns where
?id={{pageAlias:T}}
will be replaced by{{baseUrlLang:Y}}/{{pageSlug:T}}
href="[index.php]?id={{pageAlias:T}}
href='[index.php]?id={{pageAlias:T}}
- patterns for which there are no suggestions, user must fix manually
{{pageAlias:T}}
(except for those that already matched above)
- If one of the following are true then QFQ never shows this exception again:
- The search found no occurrence of any of the above patterns
- "skip check forever" link was clicked
- The check can be forced manually by setting a variable in
qfq.json
"FORCE_RUN_PAGE_SLUG_MIGRATION_CHECK":true
- The exception is shown even when no alias pattern was found
- To disable the exception the maintainer must set the variable to false or remove it
- MAYBE : Search and replace alias with slug in column
Settings.TableId
of QFQ database- If we don't do this: The occurrence of {{pageAlias:T}} as the TableSorter id will be shown in the Exception anyway. But the user will have to deal with it manually.
Technical Spec:¶
- the following columns are checked for occurrences of alias-patterns
- tt_content.bodytext
- Form.title
- Form.parameter
- Form.parameterLanguageA/B/C/D
- FormElement.label
- FormElement.note
- FormElement.value
- FormElement.sql1
- FormElement.parameter
- FormElement.parameterLanguageA/B/C/D
- The page slug for patterns containing
<alias>
will be found the following way- search
<alias>
in columnpages.slug
- when migrating from alias to slug underscore "_" was replaced by minus "-" by Typo3 automatically, so search with "_" replaced in
<alias>
- when migrating from alias to slug underscore "_" was replaced by minus "-" by Typo3 automatically, so search with "_" replaced in
- if not found search
<alias>
in columnpages.zzz_deleted_alias
if that column exists - if not found mark
<alias>
prominently as not found in error message.
- search
- If the migration was performed or skipped then the flag
T3v10AliasMigration
is added to the comment of the Form table- values
ongoing
(check is performed and exception is shown if pattern found)- Question C: is this neccessary? it's hard to implement
skip
(check is skipped)done
(check is skipped)
- The current comment looks like this:
Version=21.5.0a&functionHash=dec0567ea030ed61fe7cee8dbba2a935&functionVersion=20.11.0
- values
{{baseUrlLang:Y}}
is the baseUrl plus the language shortcut e.g./en/
- get lanugae id by
{{pageLanguage:T}}
- Unfortunatelly the language url-prefix can't be queried the same name as the id (using the Language Aspect of typo3)
- No alternative way found to get the url-prefix from Typo3
- alternative: set the language-id=>url-prefix manually in the config of qfq
- alternative: compute-url prefix by: <current-url> - {{pageSlug:T}}
- Problem: the url seen by the php script might not be the public url
- Question B : How shall we get the url-language-prefix? there is no good solution
- get lanugae id by
- Variants to check if Typo3 version is higher than 9
- Carsten found a sexy solutin: https://api.typo3.org/master/class_t_y_p_o3_1_1_c_m_s_1_1_core_1_1_utility_1_1_version_number_utility.html
- The Typo3 version might be found in a Global variable?
- if not, the symlink in the root folder could be parsed:
typo3_src -> /var/www/typo3_src-8.7.32/
- or the hacky way it is done in the password hashing function by checking if a certain Typo3 class exists
- Writing suggestions to database
- when reading QFQ code from DB remember already construct a prepared sql update statement for that piece of code
- When in replacement mode use the prepared update statement to update the database
Updated by Marc Egger almost 3 years ago
- Subject changed from T3 v10 migration script: replace id=alias patterns with id to T3 v10 migration script: replace alias-patterns
Updated by Marc Egger almost 3 years ago
Regex patterns and tests
(['"\|]p:)((?:id=)?[a-zA-Z0-9_\-]*)([&\|"']) 'p:al_ias&ssd' "p:alia-s|a98v" "p:alias"ahsdfja 'p:alias'asdj;klv "p:5&asd" 'p:54|afdg' "p:566"afdg 'p:54'asdf |p:5&asd" |p:54|afdg' |p:566"afdg |p:54'asdf |p:id=34'asdf |p:id=alias'asdf (['"\|]p:)((?:id=)?{{(?:pageAlias|pageId))(:T[0E]*}}[&\|"']) "p:{{pageAlias:T}}|avsadd" 'p:{{pageAlias:T}}&sadfsda' "p:{{pageAlias:T0}}|avsadd" "p:{{pageAlias:TE0}}|avsadd" "p:{{pageId:T0}}|avsadd" "p:{{pageId:TE0}}|avsadd" 'p:{{pageAlias:TRF0}}&sadfsda' <<< this line should not match |p:{{pageAlias:T0}}|avsadd" |p:{{pageAlias:TE0}}|avsadd" CONCAT('p:id={{pageAlias:T}}&form=facultyCR&r=', fa.id) (href=['"])((?:index\.php)?\?id=[a-zA-Z0-9_\-]*)([&"']) href="?id=alias& href='?id=al_ias& href="?id=5" href='?id=0' href="index.php?id=AlI_-AS& href='index.php?id=43-64-7& href="index.php?id=09" href='index.php?id=alias5' (href=['"])((?:index\.php)?\?id={{(?:pageAlias|pageId))(:T[0E]*}}[&"']) href="index.php?id={{pageAlias:T}}" href='index.php?id={{pageAlias:TE}}| <<< this line should not match href="?id={{pageAlias:T}}& href='?id={{pageAlias:T0}}' href="?id={{pageId:T}}& href='?id={{pageId:T0}}' href="?id={{pageAlias:T}}& href='?id={{pageAlias:T0}}' href="index.php?id={{noMatch:T}}" <<< this line should not match href="?id={{noMatch:T}}| <<< this line should not match search without suggestion: (['"\|]U:)(id=\S+?)([&\|"']) 'U:id={{printview:R}}&type=1&reId={{reId:S0}}|t:Preview|s' AS _link , '|U:id={{printview:R}}&type=1&_sip=1&reId=', re.id , '|U:id={{printview:R}}', re.id , '|U:id={{printview:R}}", re.id , 'U:id={{printview:R}}', re.id search without suggestion: (['"\|]u:\S+?[\?&])(id=\S+?)([&\|"']) 'u:index.php?id={{printview:R}}&type=1&reId={{reId:S0}}|t:Preview|s' AS _link , '|u:google.com?foo=bar&id={{printview:R}}&type=1&_sip=1&reId=', re.id , '|u:test?id={{printview:R}}', re.id , 'u:test?id={{printview:R}}', re.id , '|u:test?id={{printview:R}}", re.id search without suggestion: ({{)((?:pageAlias|pageId))(:T[BCDEFLMPRSUVY0]*}})
Updated by Marc Egger almost 3 years ago
Todo next: import T3 and QFQ Db from files in Download folder. Manually via command line, since too big. see https://stackoverflow.com/questions/11305516/how-to-import-large-sql-file-in-phpmyadmin
Todo : create a new test report with the regex tests
mysql -u megger_qfq -p megger_forkred_db < megger_forkred_db.sql mysql -u megger_qfq -p megger_forkred_t3 < megger_forkred_t3.sql
Updated by Marc Egger over 2 years ago
types of alias or id usages without suggestions:
- Page does not exist/was deleted:
'p:uploads ➡ ❎&
- Html Table dynamic id:
id="{{pageAlias ➡ ❎:T}}-form"
- Tablesorter view saver dynamic id:
{{ '{{pageAlias ➡ ❎:T}}-{{faId:R}}-cr' AS _tablesorter-view-saver }}
- javascript:
<script>window.location.replace("?id={{pageAlias ➡ ❎:T}}
- Switch on page name:
CASE ... WHEN '{{pageAlias ➡ ❎:T}}' LIKE '%stwf%' THEN 'StwF'
- Die meissten sind von diesem Typ. Falls der Slug fast gleich ist wie das alias, dann werden diese mit
{{pageSlug}}
weiterhin funktionieren.
- Die meissten sind von diesem Typ. Falls der Slug fast gleich ist wie das alias, dann werden diese mit
- Hidden input on form:
<input type="hidden" name="id" value="{{pageAlias ➡ ❎:T}}">
- Page back GET attribut in einem Link mitgegeben:
'p:upload_{{shortFkStwf:R}}&pageBack={{pageAlias ➡ ❎:T}}&
- User store variablen-name konstruiert aus pageAlias:
AND '{{{{pageAlias ➡ ❎:T}}_ShowInfoToNonAdmin:U0}}'='1'
- Record store variaben-name konstruoert aus pageAlias:
'0' AS '_={{pageAlias ➡ ❎:T}}_ShowInfoToNonAdmin'
Updated by Marc Egger over 2 years ago
TODO NEXT: special colum _link: takes slug instead of alias in first argument 'p:'TODO NEXT: {{pageSlug:T}} variabel bereitstellenTODO NEXT: find a way to compute system variable {{baseUrlLang:Y}}
Updated by Marc Egger over 2 years ago
Column As _link¶
Research¶
- important functions/constants
- TOKEN_PAGE
- TOKEN_UID
- Link>renderLink() & Report>renderColumn()
- Link>fillParameter()
- Link>buildPage()
- Link>fillParameter()
- Link>processParameter()
- what to do if 'p:' value does not start with / ?
- AbstractException.php 312 : builds p:form
- system store variabel anpassen: SYSTEM_EDIT_FORM_PAGE
- Download.php: Download ('d:...' As _link) probably doesn't work anymore due to slug
- Html2Pdf.php: probably doesn't work anymore
changes¶
- AS _link now produces absolute links with slugs
Updated by Marc Egger over 2 years ago
Todo: nach {{pageSlug:T}} muss ein fragezeichen kommen wenn & war:
https://webwork16.math.uzh.ch/megger/forkred/{{pageSlug:T}}&action=new&requesttyp=Postdoc&__dbIndexData=1
Updated by Marc Egger over 2 years ago
Updated by Marc Egger over 2 years ago
TODO NEXT: go through todos of thish ticket
Updated by Marc Egger over 2 years ago
TODO NEXT: habe die einfachen alias matches gefixed. jetzt megger/forkred testen was tut, was nicht
Updated by Marc Egger over 2 years ago
Gespraech mit Carsten:
Entscheidungen:- Definitiv Slug migration ab T3 v9. as _link funktioniert nur noch mit slugs ab v9.
- Release notes text aufsetzen.
- "migration muss gemacht werden sonst funktioniert as _link nicht mehr"
- $systemMessage feature in dev doku doukmentieren und bei variabel definition link dort hin
- T3 v8 switches einbauen
- prefix von report datei in exception angeben
- qfq.json pfad angeben in exception
- schauen ob ich in pre html tags zeilennummern anzeigen kann
- ist moeglich mit CSS aber super aufwaendig. lohnt sich nicht.
- was machen wir mit uploads page in forkred? > wahrscheinlich bei Simon UZH mob passiert
- Hinweis geben, dass ein QFQ tt-content deaktiviert ist oder die page darauf. "tt-content is probably not in use"
- andere QFQ installationen migrieren und testen
Updated by Marc Egger over 2 years ago
TODO NEXT: Remove Carsten fixes using BASE_DIR_FAKE. Now fix everything.
command to run phpunit:
www-data@webwork16:/var/www/html/megger/forkred/typo3conf/ext/qfq$ vendor/bin/phpunit --configuration phpunit.xml
Remove Dynamic SQL before running unittests! Do this by adding following to qfq.json:
"fillStoreSystemBySql1": "", "fillStoreSystemBySql2": "", "fillStoreSystemBySql3": ""
Updated by Marc Egger over 2 years ago
tested if this version of QFQ still works on Typo3 version 8.
Test instanz: webwork16/megger/lehrkredit8
Updated by Marc Egger over 2 years ago
Completed the slug migration in megger/forkred with some hacks.
E.g. manual backlinks which used pageAlias now point to the page "broken"
also dynamically generated variable names using pageAlias now use pageSlug, but this probably wont work.
TODO NEXT : FK: Find way to fix pdf genartion link. FK> Administration FK > PDF > PDF TODO NEXT : Consulting backsync auf Dev, upgrade T3 9, alias migration
Updated by Marc Egger over 2 years ago
Created ticket to update Consulting. #12951
LAST DONE: fixed form redirect url with slug
TODO NEXT : update QFQ version in Consulting and test form redirectTODO NEXT : fix Form edit button, see todo aboveTODO NEXT : nach konstante INDEX_PHP suchen
Updated by Marc Egger over 2 years ago
DID LAST: added id=... regex pattern and installed new QFQ in Consulting. now get new occurrences of alias.
TODO NEXT : fix alias occurrences in Consulting and test all.
Updated by Marc Egger over 2 years ago
TODO NEXT : make sure that print.php takes slug instead of id/alias. see Doku: "QFQ implements a small PHP wrapper" . Also fix it in Doku!TODO NEXT : search fo pageId in all Dokumentation
further todos in user doku:
print.php in doku und echt fixen: print.php?id={current pageId} ... Different browser prints the same page in different variations. To prevent this, QFQ implements a small PHP wrapper
`print.php` with uses `wkhtmltopdf` to convert HTML to PDF.Fix Tutorial.rstlook at all changes made in doku and see if table is brokenAS _link usw. parameter dokumentation anpassen p: U: usw.alle p:id beispiele usw. ersetzen (suche nach "p:")alle beispiele die {{pageAlias}} oder so benutzen mit {{pageSlug}} fixen (suche nach "pageAlias" und "pageId")
Updated by Marc Egger over 2 years ago
- Status changed from In Progress to Feedback
- Assignee changed from Marc Egger to Carsten Rose
Die Slug Migration ist fertig und die Dokumentation ist angepasst.
Sie ist in folgendem Branch implementiert: F12584T3V10MigrationScriptReplaceAliasPatterns
Informationen sind in Release.rst eingetrage und koennen beim erstellen der naechsten release notes verwendet werden.
Die aktuelle Version von diesem Branch ist auf webwork16 consulting/dev installiert und funktioniert. In dem Ticket #12951 ist der gesammte upgrade prozess fuer Consulting dev dokumentiert.
Updated by Carsten Rose almost 2 years ago
- Subject changed from T3 v10 migration script: replace alias-patterns to T3 v10 migration script: replace alias-patterns (v11)
Updated by Carsten Rose over 1 year ago
- Target version changed from next3 to 22.10.1
- Prio Planung set to No