Project

General

Profile

Actions

Feature #12584

closed

Feature #12440: Typo3 V10 upgrade (durchfuehren und testen)

T3 v10 migration script: replace alias-patterns (v11)

Added by Marc Egger almost 3 years ago. Updated over 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Carsten Rose
Target version:
Start date:
25.05.2021
Due date:
% Done:

100%

Estimated time:
Discuss:
Prio Planung:
No
Vote:

Description

ALTE VERSION von Spezifikation! neue Version im Kommentar

Functional Spec:

  • If in any Report or Form there is an occurrence of an "id=alias" pattern (see bellow) and the T3 version is higher than 9, then QFQ shows an exception with the following content:
    • link to automatically apply all suggested replacements
    • link to reload and skip this check and exception forever
    • Detailed list of each occurrence together with the suggested replacement
    • If no suggested id was found for an alias then this is marked prominently in the error message together with the location of the occurrence
  • patterns where <alias> will be replaced by id
    • href="?id=<alias>&
    • href='?id=<alias>&
    • href="?id=<alias>"
    • href='?id=<alias>'
  • patterns where {{pageAlias:T}} will be replaced by {{pageId:T}}
    • href="?id={{pageAlias:T}}
    • href='?id={{pageAlias:T}}
  • If one of the following are true then QFQ never shows this exception again:
    • The search found no occurrence of any of the above patterns
    • "skip check forever" link was clicked
  • Question A : should all occurrences of {{pageAlias:T}} be replaced by {{pageId:T}} everywhere even when there is no "href"?
    • The page alias will not have any meaning or won't even exist for Typo3 v10 upwards
    • alternatively we could introduce a new variable {{pagePath:T}}. But I don't know how easy it will be to compute that path reliably and I would rather not ask Typo3 to do it...

Technical Spec:

  • the following columns are checked for occurrences of `id=alias`
    • tt_content.bodytext
    • Form.title
    • Form.parameter
    • Form.parameterLanguageA/B/C/D
    • FormElement.label
    • FormElement.note
    • FormElement.value
    • FormElement.sql1
    • FormElement.parameter
    • FormElement.parameterLanguageA/B/C/D
  • The page id for patterns containing <alias> will be found the following way
    • search <alias> in column pages.slug
      • when migrating from alias to slug underscore "_" was replaced by minus "-" by Typo3 automatically, so search again with "_" replaced in <alias>
    • if not found, search in column pages.zzz_deleted_alias if that column exists
    • if not found mark <alias> prominently as not found in error message.
  • If the migration was performed or skipped then the flag skipAliasMigration is added to the comment of the Form table
    • The current comment looks like this: Version=21.5.0a&functionHash=dec0567ea030ed61fe7cee8dbba2a935&functionVersion=20.11.0
  • Variants to check if Typo3 version is higher than 9
    • The Typo3 version might be found in a Global variable?
    • if not, the symlink in the root folder could be parsed: typo3_src -> /var/www/typo3_src-8.7.32/
    • or the hacky way it is done in the password hashing function by checking if a certain Typo3 class exists
Actions #1

Updated by Marc Egger almost 3 years ago

  • Description updated (diff)
Actions #2

Updated by Marc Egger almost 3 years ago

  • Description updated (diff)
Actions #3

Updated by Marc Egger almost 3 years ago

  • Description updated (diff)
Actions #4

Updated by Marc Egger almost 3 years ago

  • Description updated (diff)
Actions #5

Updated by Marc Egger almost 3 years ago

  • Description updated (diff)
Actions #6

Updated by Marc Egger almost 3 years ago

  • Description updated (diff)
Actions #7

Updated by Marc Egger almost 3 years ago

  • Description updated (diff)
Actions #8

Updated by Marc Egger almost 3 years ago

  • Description updated (diff)
Actions #9

Updated by Marc Egger almost 3 years ago

  • Description updated (diff)
Actions #10

Updated by Marc Egger almost 3 years ago

  • Subject changed from T3 v10 migration script: replace all id=alias with id to T3 v10 migration script: replace id=alias patterns with id
Actions #11

Updated by Marc Egger almost 3 years ago

  • Description updated (diff)
Actions #12

Updated by Marc Egger almost 3 years ago

  • Status changed from New to Feedback
  • Assignee changed from Marc Egger to Carsten Rose

Bitte um feadback und deine Meinung zu "Question A"

Actions #13

Updated by Marc Egger almost 3 years ago

  • Parent task set to #12440
Actions #14

Updated by Marc Egger almost 3 years ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Carsten Rose to Marc Egger
Actions #15

Updated by Marc Egger almost 3 years ago

  • Description updated (diff)

neue Version nach Besprechung Carsten (danach nicht mehr aktuallisiert, siehe Release.rst fuer aktuelle Infos)

General

  • alias wird bei der Migration komplett mit slug ersetzt
  • nach der Migration wird davon ausgegangen, dass nur noch slug verwendet wird

Functional Spec:

  • If in any Report or Form there is an occurrence of an "alias-pattern" (see bellow) and the T3 version is higher than 9, then QFQ shows an exception with the following content:
    • link to automatically apply all suggested replacements
    • link to reload and skip this check and exception forever
    • Detailed list of each occurrence together with the suggested replacement
    • If no suggestion was found for a pattern then this is marked prominently in the error message together with the location of the occurrence
  • patterns where <alias/id> will be replaced by <slug>
    • p:<alias/id>&
    • p:<alias/id>|
    • "p:<alias/id>"
    • 'p:<alias/id>'
    • |p:<alias/id>'
    • |p:id=<alias/id>'
  • patterns where {{pageAlias:T}} will be replaced by {{pageSlug:T}}
    • "p:{{pageAlias:T}}
    • 'p:{{pageAlias:T}}
    • |p:{{pageAlias:T}}
    • 'p:id={{pageAlias:T}}
  • patterns where <alias> will be replaced by {{baseUrlLang:Y}}/<slug>
    • href="[index.php]?id=<alias/id>&
    • href='[index.php]?id=<alias/id>&
    • href="[index.php]?id=<alias/id>"
    • href='[index.php]?id=<alias/id>'
  • patterns where ?id={{pageAlias:T}} will be replaced by {{baseUrlLang:Y}}/{{pageSlug:T}}
    • href="[index.php]?id={{pageAlias:T}}
    • href='[index.php]?id={{pageAlias:T}}
  • patterns for which there are no suggestions, user must fix manually
    • {{pageAlias:T}} (except for those that already matched above)
  • If one of the following are true then QFQ never shows this exception again:
    • The search found no occurrence of any of the above patterns
    • "skip check forever" link was clicked
  • The check can be forced manually by setting a variable in qfq.json
    • "FORCE_RUN_PAGE_SLUG_MIGRATION_CHECK":true
    • The exception is shown even when no alias pattern was found
    • To disable the exception the maintainer must set the variable to false or remove it
  • MAYBE : Search and replace alias with slug in column Settings.TableId of QFQ database
    • If we don't do this: The occurrence of {{pageAlias:T}} as the TableSorter id will be shown in the Exception anyway. But the user will have to deal with it manually.

Technical Spec:

  • the following columns are checked for occurrences of alias-patterns
    • tt_content.bodytext
    • Form.title
    • Form.parameter
    • Form.parameterLanguageA/B/C/D
    • FormElement.label
    • FormElement.note
    • FormElement.value
    • FormElement.sql1
    • FormElement.parameter
    • FormElement.parameterLanguageA/B/C/D
  • The page slug for patterns containing <alias> will be found the following way
    • search <alias> in column pages.slug
      • when migrating from alias to slug underscore "_" was replaced by minus "-" by Typo3 automatically, so search with "_" replaced in <alias>
    • if not found search <alias> in column pages.zzz_deleted_alias if that column exists
    • if not found mark <alias> prominently as not found in error message.
  • If the migration was performed or skipped then the flag T3v10AliasMigration is added to the comment of the Form table
    • values
      • ongoing (check is performed and exception is shown if pattern found)
        • Question C: is this neccessary? it's hard to implement
      • skip (check is skipped)
      • done (check is skipped)
    • The current comment looks like this: Version=21.5.0a&functionHash=dec0567ea030ed61fe7cee8dbba2a935&functionVersion=20.11.0
  • {{baseUrlLang:Y}} is the baseUrl plus the language shortcut e.g. /en/
    • get lanugae id by {{pageLanguage:T}}
    • Unfortunatelly the language url-prefix can't be queried the same name as the id (using the Language Aspect of typo3)
    • No alternative way found to get the url-prefix from Typo3
    • alternative: set the language-id=>url-prefix manually in the config of qfq
    • alternative: compute-url prefix by: <current-url> - {{pageSlug:T}}
      • Problem: the url seen by the php script might not be the public url
    • Question B : How shall we get the url-language-prefix? there is no good solution
  • Writing suggestions to database
    • when reading QFQ code from DB remember already construct a prepared sql update statement for that piece of code
    • When in replacement mode use the prepared update statement to update the database
Actions #16

Updated by Marc Egger almost 3 years ago

  • Subject changed from T3 v10 migration script: replace id=alias patterns with id to T3 v10 migration script: replace alias-patterns
Actions #17

Updated by Marc Egger almost 3 years ago

Regex patterns and tests

(['"\|]p:)((?:id=)?[a-zA-Z0-9_\-]*)([&\|"'])
'p:al_ias&ssd'
"p:alia-s|a98v" 
"p:alias"ahsdfja
'p:alias'asdj;klv
"p:5&asd" 
'p:54|afdg'
"p:566"afdg
'p:54'asdf
|p:5&asd" 
|p:54|afdg'
|p:566"afdg
|p:54'asdf
|p:id=34'asdf
|p:id=alias'asdf

(['"\|]p:)((?:id=)?{{(?:pageAlias|pageId))(:T[0E]*}}[&\|"'])
"p:{{pageAlias:T}}|avsadd" 
'p:{{pageAlias:T}}&sadfsda'
"p:{{pageAlias:T0}}|avsadd" 
"p:{{pageAlias:TE0}}|avsadd" 
"p:{{pageId:T0}}|avsadd" 
"p:{{pageId:TE0}}|avsadd" 
'p:{{pageAlias:TRF0}}&sadfsda'  <<< this line should not match
|p:{{pageAlias:T0}}|avsadd" 
|p:{{pageAlias:TE0}}|avsadd" 
CONCAT('p:id={{pageAlias:T}}&form=facultyCR&r=', fa.id)

(href=['"])((?:index\.php)?\?id=[a-zA-Z0-9_\-]*)([&"'])
href="?id=alias&
href='?id=al_ias&
href="?id=5" 
href='?id=0'
href="index.php?id=AlI_-AS&
href='index.php?id=43-64-7&
href="index.php?id=09" 
href='index.php?id=alias5'

(href=['"])((?:index\.php)?\?id={{(?:pageAlias|pageId))(:T[0E]*}}[&"'])
href="index.php?id={{pageAlias:T}}" 
href='index.php?id={{pageAlias:TE}}|  <<< this line should not match
href="?id={{pageAlias:T}}&
href='?id={{pageAlias:T0}}'
href="?id={{pageId:T}}&
href='?id={{pageId:T0}}'
href="?id={{pageAlias:T}}&
href='?id={{pageAlias:T0}}'
href="index.php?id={{noMatch:T}}"  <<< this line should not match
href="?id={{noMatch:T}}|  <<< this line should not match

search without suggestion:
(['"\|]U:)(id=\S+?)([&\|"'])
'U:id={{printview:R}}&type=1&reId={{reId:S0}}|t:Preview|s' AS _link
, '|U:id={{printview:R}}&type=1&_sip=1&reId=', re.id
, '|U:id={{printview:R}}', re.id
, '|U:id={{printview:R}}", re.id
, 'U:id={{printview:R}}', re.id

search without suggestion:
(['"\|]u:\S+?[\?&])(id=\S+?)([&\|"'])
'u:index.php?id={{printview:R}}&type=1&reId={{reId:S0}}|t:Preview|s' AS _link
, '|u:google.com?foo=bar&id={{printview:R}}&type=1&_sip=1&reId=', re.id
, '|u:test?id={{printview:R}}', re.id
, 'u:test?id={{printview:R}}', re.id
, '|u:test?id={{printview:R}}", re.id

search without suggestion:
({{)((?:pageAlias|pageId))(:T[BCDEFLMPRSUVY0]*}})
Actions #18

Updated by Marc Egger almost 3 years ago

Todo next: import T3 and QFQ Db from files in Download folder. Manually via command line, since too big. see https://stackoverflow.com/questions/11305516/how-to-import-large-sql-file-in-phpmyadmin

Todo : create a new test report with the regex tests

mysql -u megger_qfq -p megger_forkred_db < megger_forkred_db.sql 
mysql -u megger_qfq -p megger_forkred_t3 < megger_forkred_t3.sql
Actions #19

Updated by Marc Egger over 2 years ago

types of alias or id usages without suggestions:

  • Page does not exist/was deleted: 'p:uploads ➡ ❎&
  • Html Table dynamic id: id="{{pageAlias ➡ ❎:T}}-form"
  • Tablesorter view saver dynamic id: {{ '{{pageAlias ➡ ❎:T}}-{{faId:R}}-cr' AS _tablesorter-view-saver }}
  • javascript: <script>window.location.replace("?id={{pageAlias ➡ ❎:T}}
  • Switch on page name: CASE ... WHEN '{{pageAlias ➡ ❎:T}}' LIKE '%stwf%' THEN 'StwF'
    • Die meissten sind von diesem Typ. Falls der Slug fast gleich ist wie das alias, dann werden diese mit {{pageSlug}} weiterhin funktionieren.
  • Hidden input on form: <input type="hidden" name="id" value="{{pageAlias ➡ ❎:T}}">
  • Page back GET attribut in einem Link mitgegeben: 'p:upload_{{shortFkStwf:R}}&pageBack={{pageAlias ➡ ❎:T}}&
  • User store variablen-name konstruiert aus pageAlias: AND '{{{{pageAlias ➡ ❎:T}}_ShowInfoToNonAdmin:U0}}'='1'
  • Record store variaben-name konstruoert aus pageAlias: '0' AS '_={{pageAlias ➡ ❎:T}}_ShowInfoToNonAdmin'
Actions #20

Updated by Marc Egger over 2 years ago

TODO NEXT: special colum _link: takes slug instead of alias in first argument 'p:'
TODO NEXT: {{pageSlug:T}} variabel bereitstellen
TODO NEXT: find a way to compute system variable {{baseUrlLang:Y}}

Actions #21

Updated by Marc Egger over 2 years ago

Column As _link

Research

  • important functions/constants
    • TOKEN_PAGE
    • TOKEN_UID
    • Link>renderLink() & Report>renderColumn()
      • Link>fillParameter()
        • Link>buildPage()
    • Link>processParameter()
  • what to do if 'p:' value does not start with / ?
    • AbstractException.php 312 : builds p:form
    • system store variabel anpassen: SYSTEM_EDIT_FORM_PAGE
  • Download.php: Download ('d:...' As _link) probably doesn't work anymore due to slug
  • Html2Pdf.php: probably doesn't work anymore

changes

  • AS _link now produces absolute links with slugs
Actions #22

Updated by Marc Egger over 2 years ago

Todo: nach {{pageSlug:T}} muss ein fragezeichen kommen wenn & war:

https://webwork16.math.uzh.ch/megger/forkred/{{pageSlug:T}}&action=new&requesttyp=Postdoc&__dbIndexData=1
Actions #23

Updated by Marc Egger over 2 years ago

provide {{pageSlug:T}}

todo

  • replace all occurrences of TYPO3_PAGE_ALIAS in code >> partially done
  • what about {{pageId}} ? and TYPO3_PAGE_ID in code ? >> partially replaced, id=<number> will still work in T3 10
Actions #24

Updated by Marc Egger over 2 years ago

TODO NEXT: go through todos of thish ticket

Actions #25

Updated by Marc Egger over 2 years ago

TODO NEXT: habe die einfachen alias matches gefixed. jetzt megger/forkred testen was tut, was nicht

Actions #26

Updated by Marc Egger over 2 years ago

Gespraech mit Carsten:

Entscheidungen:
  • Definitiv Slug migration ab T3 v9. as _link funktioniert nur noch mit slugs ab v9.
todo (bereits in ticket uebertragen):
  • Release notes text aufsetzen.
    • "migration muss gemacht werden sonst funktioniert as _link nicht mehr"
  • $systemMessage feature in dev doku doukmentieren und bei variabel definition link dort hin
  • T3 v8 switches einbauen
  • prefix von report datei in exception angeben
  • qfq.json pfad angeben in exception
  • schauen ob ich in pre html tags zeilennummern anzeigen kann
    • ist moeglich mit CSS aber super aufwaendig. lohnt sich nicht.
  • was machen wir mit uploads page in forkred? > wahrscheinlich bei Simon UZH mob passiert
  • Hinweis geben, dass ein QFQ tt-content deaktiviert ist oder die page darauf. "tt-content is probably not in use"
  • andere QFQ installationen migrieren und testen
Actions #27

Updated by Marc Egger over 2 years ago

TODO NEXT: todos mit A machen

Actions #28

Updated by Marc Egger over 2 years ago

TODO NEXT: Remove Carsten fixes using BASE_DIR_FAKE. Now fix everything.

command to run phpunit:

www-data@webwork16:/var/www/html/megger/forkred/typo3conf/ext/qfq$ vendor/bin/phpunit --configuration phpunit.xml

Remove Dynamic SQL before running unittests! Do this by adding following to qfq.json:

"fillStoreSystemBySql1": "",
"fillStoreSystemBySql2": "",
"fillStoreSystemBySql3": "" 
Actions #29

Updated by Marc Egger over 2 years ago

tested if this version of QFQ still works on Typo3 version 8.
Test instanz: webwork16/megger/lehrkredit8

Actions #30

Updated by Marc Egger over 2 years ago

Completed the slug migration in megger/forkred with some hacks.

E.g. manual backlinks which used pageAlias now point to the page "broken"

also dynamically generated variable names using pageAlias now use pageSlug, but this probably wont work.

TODO NEXT : FK: Find way to fix pdf genartion link. FK> Administration FK > PDF > PDF
TODO NEXT : Consulting backsync auf Dev, upgrade T3 9, alias migration

Actions #31

Updated by Marc Egger over 2 years ago

Created ticket to update Consulting. #12951

LAST DONE: fixed form redirect url with slug

TODO NEXT : update QFQ version in Consulting and test form redirect
TODO NEXT : fix Form edit button, see todo above
TODO NEXT : nach konstante INDEX_PHP suchen

Actions #32

Updated by Marc Egger over 2 years ago

DID LAST: added id=... regex pattern and installed new QFQ in Consulting. now get new occurrences of alias.

TODO NEXT : fix alias occurrences in Consulting and test all.

Actions #33

Updated by Marc Egger over 2 years ago

  • Description updated (diff)
Actions #34

Updated by Marc Egger over 2 years ago

TODO NEXT : make sure that print.php takes slug instead of id/alias. see Doku: "QFQ implements a small PHP wrapper" . Also fix it in Doku!
TODO NEXT : search fo pageId in all Dokumentation

further todos in user doku:

  • print.php in doku und echt fixen: print.php?id={current pageId} ... Different browser prints the same page in different variations. To prevent this, QFQ implements a small PHP wrapper
    `print.php` with uses `wkhtmltopdf` to convert HTML to PDF.
  • Fix Tutorial.rst
  • look at all changes made in doku and see if table is broken
  • AS _link usw. parameter dokumentation anpassen p: U: usw.
  • alle p:id beispiele usw. ersetzen (suche nach "p:")
  • alle beispiele die {{pageAlias}} oder so benutzen mit {{pageSlug}} fixen (suche nach "pageAlias" und "pageId")
Actions #35

Updated by Marc Egger over 2 years ago

  • Status changed from In Progress to Feedback
  • Assignee changed from Marc Egger to Carsten Rose

Die Slug Migration ist fertig und die Dokumentation ist angepasst.

Sie ist in folgendem Branch implementiert: F12584T3V10MigrationScriptReplaceAliasPatterns

Informationen sind in Release.rst eingetrage und koennen beim erstellen der naechsten release notes verwendet werden.

Die aktuelle Version von diesem Branch ist auf webwork16 consulting/dev installiert und funktioniert. In dem Ticket #12951 ist der gesammte upgrade prozess fuer Consulting dev dokumentiert.

Actions #36

Updated by Carsten Rose almost 2 years ago

  • Subject changed from T3 v10 migration script: replace alias-patterns to T3 v10 migration script: replace alias-patterns (v11)
Actions #37

Updated by Carsten Rose over 1 year ago

  • Target version changed from next3 to 22.10.1
  • Prio Planung set to No
Actions #38

Updated by Carsten Rose over 1 year ago

  • Status changed from Feedback to Closed
Actions

Also available in: Atom PDF