Project

General

Profile

Feature #12584

Feature #12440: Typo3 V10 upgrade Testen

T3 v10 migration script: replace alias-patterns

Added by Marc Egger 2 months ago. Updated 15 days ago.

Status:
In Progress
Priority:
Normal
Assignee:
Target version:
Start date:
25.05.2021
Due date:
% Done:

60%

Estimated time:
Discuss:

Description

ALTE VERSION, neue Version im Kommentar

Functional Spec:

  • If in any Report or Form there is an occurrence of an "id=alias" pattern (see bellow) and the T3 version is higher than 9, then QFQ shows an exception with the following content:
    • link to automatically apply all suggested replacements
    • link to reload and skip this check and exception forever
    • Detailed list of each occurrence together with the suggested replacement
    • If no suggested id was found for an alias then this is marked prominently in the error message together with the location of the occurrence
  • patterns where <alias> will be replaced by id
    • href="?id=<alias>&
    • href='?id=<alias>&
    • href="?id=<alias>"
    • href='?id=<alias>'
  • patterns where {{pageAlias:T}} will be replaced by {{pageId:T}}
    • href="?id={{pageAlias:T}}
    • href='?id={{pageAlias:T}}
  • If one of the following are true then QFQ never shows this exception again:
    • The search found no occurrence of any of the above patterns
    • "skip check forever" link was clicked
  • Question A : should all occurrences of {{pageAlias:T}} be replaced by {{pageId:T}} everywhere even when there is no "href"?
    • The page alias will not have any meaning or won't even exist for Typo3 v10 upwards
    • alternatively we could introduce a new variable {{pagePath:T}}. But I don't know how easy it will be to compute that path reliably and I would rather not ask Typo3 to do it...

Technical Spec:

  • the following columns are checked for occurrences of `id=alias`
    • tt_content.bodytext
    • Form.title
    • Form.parameter
    • Form.parameterLanguageA/B/C/D
    • FormElement.label
    • FormElement.note
    • FormElement.value
    • FormElement.sql1
    • FormElement.parameter
    • FormElement.parameterLanguageA/B/C/D
  • The page id for patterns containing <alias> will be found the following way
    • search <alias> in column pages.slug
      • when migrating from alias to slug underscore "_" was replaced by minus "-" by Typo3 automatically, so search again with "_" replaced in <alias>
    • if not found, search in column pages.zzz_deleted_alias if that column exists
    • if not found mark <alias> prominently as not found in error message.
  • If the migration was performed or skipped then the flag skipAliasMigration is added to the comment of the Form table
    • The current comment looks like this: Version=21.5.0a&functionHash=dec0567ea030ed61fe7cee8dbba2a935&functionVersion=20.11.0
  • Variants to check if Typo3 version is higher than 9
    • The Typo3 version might be found in a Global variable?
    • if not, the symlink in the root folder could be parsed: typo3_src -> /var/www/typo3_src-8.7.32/
    • or the hacky way it is done in the password hashing function by checking if a certain Typo3 class exists
#1

Updated by Marc Egger 2 months ago

  • Description updated (diff)
#2

Updated by Marc Egger 2 months ago

  • Description updated (diff)
#3

Updated by Marc Egger 2 months ago

  • Description updated (diff)
#4

Updated by Marc Egger 2 months ago

  • Description updated (diff)
#5

Updated by Marc Egger 2 months ago

  • Description updated (diff)
#6

Updated by Marc Egger 2 months ago

  • Description updated (diff)
#7

Updated by Marc Egger 2 months ago

  • Description updated (diff)
#8

Updated by Marc Egger 2 months ago

  • Description updated (diff)
#9

Updated by Marc Egger 2 months ago

  • Description updated (diff)
#10

Updated by Marc Egger 2 months ago

  • Subject changed from T3 v10 migration script: replace all id=alias with id to T3 v10 migration script: replace id=alias patterns with id
#11

Updated by Marc Egger 2 months ago

  • Description updated (diff)
#12

Updated by Marc Egger 2 months ago

  • Status changed from New to Feedback
  • Assignee changed from Marc Egger to Carsten Rose

Bitte um feadback und deine Meinung zu "Question A"

#13

Updated by Marc Egger 2 months ago

  • Parent task set to #12440
#14

Updated by Marc Egger 2 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Carsten Rose to Marc Egger
#15

Updated by Marc Egger 2 months ago

  • Description updated (diff)

neue Version nach Besprechung Carsten

General

  • alias wird bei der Migration komplett mit slug ersetzt
  • nach der Migration wird davon ausgegangen, dass nur noch slug verwendet wird

Functional Spec:

  • If in any Report or Form there is an occurrence of an "alias-pattern" (see bellow) and the T3 version is higher than 9, then QFQ shows an exception with the following content:
    • link to automatically apply all suggested replacements
    • link to reload and skip this check and exception forever
    • Detailed list of each occurrence together with the suggested replacement
    • If no suggestion was found for a pattern then this is marked prominently in the error message together with the location of the occurrence
  • patterns where <alias/id> will be replaced by <slug>
    • p:<alias/id>&
    • p:<alias/id>|
    • "p:<alias/id>"
    • 'p:<alias/id>'
    • |p:<alias/id>'
    • |p:id=<alias/id>'
  • patterns where {{pageAlias:T}} will be replaced by {{pageSlug:T}}
    • "p:{{pageAlias:T}}
    • 'p:{{pageAlias:T}}
    • |p:{{pageAlias:T}}
    • 'p:id={{pageAlias:T}}
  • patterns where <alias> will be replaced by {{baseUrlLang:Y}}/<slug>
    • href="[index.php]?id=<alias/id>&
    • href='[index.php]?id=<alias/id>&
    • href="[index.php]?id=<alias/id>"
    • href='[index.php]?id=<alias/id>'
  • patterns where ?id={{pageAlias:T}} will be replaced by {{baseUrlLang:Y}}/{{pageSlug:T}}
    • href="[index.php]?id={{pageAlias:T}}
    • href='[index.php]?id={{pageAlias:T}}
  • patterns for which there are no suggestions, user must fix manually
    • {{pageAlias:T}} (except for those that already matched above)
  • If one of the following are true then QFQ never shows this exception again:
    • The search found no occurrence of any of the above patterns
    • "skip check forever" link was clicked
  • The check can be forced manually by setting a variable in qfq.json
    • "FORCE_RUN_PAGE_SLUG_MIGRATION_CHECK":true
    • The exception is shown even when no alias pattern was found
    • To disable the exception the maintainer must set the variable to false or remove it
  • MAYBE : Search and replace alias with slug in column Settings.TableId of QFQ database
    • If we don't do this: The occurrence of {{pageAlias:T}} as the TableSorter id will be shown in the Exception anyway. But the user will have to deal with it manually.

Technical Spec:

  • the following columns are checked for occurrences of alias-patterns
    • tt_content.bodytext
    • Form.title
    • Form.parameter
    • Form.parameterLanguageA/B/C/D
    • FormElement.label
    • FormElement.note
    • FormElement.value
    • FormElement.sql1
    • FormElement.parameter
    • FormElement.parameterLanguageA/B/C/D
  • The page slug for patterns containing <alias> will be found the following way
    • search <alias> in column pages.slug
      • when migrating from alias to slug underscore "_" was replaced by minus "-" by Typo3 automatically, so search with "_" replaced in <alias>
    • if not found search <alias> in column pages.zzz_deleted_alias if that column exists
    • if not found mark <alias> prominently as not found in error message.
  • If the migration was performed or skipped then the flag T3v10AliasMigration is added to the comment of the Form table
    • values
      • ongoing (check is performed and exception is shown if pattern found)
        • Question C: is this neccessary? it's hard to implement
      • skip (check is skipped)
      • done (check is skipped)
    • The current comment looks like this: Version=21.5.0a&functionHash=dec0567ea030ed61fe7cee8dbba2a935&functionVersion=20.11.0
  • {{baseUrlLang:Y}} is the baseUrl plus the language shortcut e.g. /en/
    • get lanugae id by {{pageLanguage:T}}
    • Unfortunatelly the language url-prefix can't be queried the same name as the id (using the Language Aspect of typo3)
    • No alternative way found to get the url-prefix from Typo3
    • alternative: set the language-id=>url-prefix manually in the config of qfq
    • alternative: compute-url prefix by: <current-url> - {{pageSlug:T}}
      • Problem: the url seen by the php script might not be the public url
    • Question B : How shall we get the url-language-prefix? there is no good solution
  • Writing suggestions to database
    • when reading QFQ code from DB remember already construct a prepared sql update statement for that piece of code
    • When in replacement mode use the prepared update statement to update the database
#16

Updated by Marc Egger 2 months ago

  • Subject changed from T3 v10 migration script: replace id=alias patterns with id to T3 v10 migration script: replace alias-patterns
#17

Updated by Marc Egger about 2 months ago

Regex patterns and tests

(['"\|]p:)((?:id=)?[a-zA-Z0-9_\-]*)([&\|"'])
'p:al_ias&ssd'
"p:alia-s|a98v" 
"p:alias"ahsdfja
'p:alias'asdj;klv
"p:5&asd" 
'p:54|afdg'
"p:566"afdg
'p:54'asdf
|p:5&asd" 
|p:54|afdg'
|p:566"afdg
|p:54'asdf
|p:id=34'asdf
|p:id=alias'asdf

(['"\|]p:)((?:id=)?{{(?:pageAlias|pageId))(:T[0E]*}}[&\|"'])
"p:{{pageAlias:T}}|avsadd" 
'p:{{pageAlias:T}}&sadfsda'
"p:{{pageAlias:T0}}|avsadd" 
"p:{{pageAlias:TE0}}|avsadd" 
"p:{{pageId:T0}}|avsadd" 
"p:{{pageId:TE0}}|avsadd" 
'p:{{pageAlias:TRF0}}&sadfsda'  <<< this line should not match
|p:{{pageAlias:T0}}|avsadd" 
|p:{{pageAlias:TE0}}|avsadd" 
CONCAT('p:id={{pageAlias:T}}&form=facultyCR&r=', fa.id)

(href=['"])((?:index\.php)?\?id=[a-zA-Z0-9_\-]*)([&"'])
href="?id=alias&
href='?id=al_ias&
href="?id=5" 
href='?id=0'
href="index.php?id=AlI_-AS&
href='index.php?id=43-64-7&
href="index.php?id=09" 
href='index.php?id=alias5'

(href=['"])((?:index\.php)?\?id={{(?:pageAlias|pageId))(:T[0E]*}}[&"'])
href="index.php?id={{pageAlias:T}}" 
href='index.php?id={{pageAlias:TE}}|  <<< this line should not match
href="?id={{pageAlias:T}}&
href='?id={{pageAlias:T0}}'
href="?id={{pageId:T}}&
href='?id={{pageId:T0}}'
href="?id={{pageAlias:T}}&
href='?id={{pageAlias:T0}}'
href="index.php?id={{noMatch:T}}"  <<< this line should not match
href="?id={{noMatch:T}}|  <<< this line should not match

search without suggestion:
({{)((?:pageAlias|pageId))(:T[BCDEFLMPRSUVY0]*}})
#18

Updated by Marc Egger about 2 months ago

Todo next: import T3 and QFQ Db from files in Download folder. Manually via command line, since too big. see https://stackoverflow.com/questions/11305516/how-to-import-large-sql-file-in-phpmyadmin

Todo : create a new test report with the regex tests

mysql -u megger_qfq -p megger_forkred_db < megger_forkred_db.sql 
mysql -u megger_qfq -p megger_forkred_t3 < megger_forkred_t3.sql
#19

Updated by Marc Egger about 1 month ago

types of alias or id usages without suggestions:

  • Page does not exist/was deleted: 'p:uploads ➡ ❎&
  • Html Table dynamic id: id="{{pageAlias ➡ ❎:T}}-form"
  • Tablesorter view saver dynamic id: {{ '{{pageAlias ➡ ❎:T}}-{{faId:R}}-cr' AS _tablesorter-view-saver }}
  • javascript: <script>window.location.replace("?id={{pageAlias ➡ ❎:T}}
  • Switch on page name: CASE ... WHEN '{{pageAlias ➡ ❎:T}}' LIKE '%stwf%' THEN 'StwF'
    • Die meissten sind von diesem Typ. Falls der Slug fast gleich ist wie das alias, dann werden diese mit {{pageSlug}} weiterhin funktionieren.
  • Hidden input on form: <input type="hidden" name="id" value="{{pageAlias ➡ ❎:T}}">
  • Page back GET attribut in einem Link mitgegeben: 'p:upload_{{shortFkStwf:R}}&pageBack={{pageAlias ➡ ❎:T}}&
  • User store variablen-name konstruiert aus pageAlias: AND '{{{{pageAlias ➡ ❎:T}}_ShowInfoToNonAdmin:U0}}'='1'
  • Record store variaben-name konstruoert aus pageAlias: '0' AS '_={{pageAlias ➡ ❎:T}}_ShowInfoToNonAdmin'
#20

Updated by Marc Egger about 1 month ago

TODO NEXT: special colum _link: takes slug instead of alias in first argument 'p:'
TODO NEXT: {{pageSlug:T}} variabel bereitstellen
TODO NEXT: find a way to compute system variable {{baseUrlLang:Y}}

#21

Updated by Marc Egger about 1 month ago

Column As _link

Research

  • important functions/constants
    • TOKEN_PAGE
    • TOKEN_UID
    • Link>renderLink() & Report>renderColumn()
      • Link>fillParameter()
        • Link>buildPage()
    • Link>processParameter()
  • what to do if 'p:' value does not start with / ?
    • AbstractException.php 312 : builds p:form
    • system store variabel anpassen: SYSTEM_EDIT_FORM_PAGE
  • Download.php: Download ('d:...' As _link) probably doesn't work anymore due to slug
  • Html2Pdf.php: probably doesn't work anymore

changes

  • AS _link now produces absolute links with slugs
#22

Updated by Marc Egger about 1 month ago

Todo: nach {{pageSlug:T}} muss ein fragezeichen kommen wenn & war:

https://webwork16.math.uzh.ch/megger/forkred/{{pageSlug:T}}&action=new&requesttyp=Postdoc&__dbIndexData=1
#23

Updated by Marc Egger about 1 month ago

provide {{pageSlug:T}}

todo

  • replace all occurrences of TYPO3_PAGE_ALIAS in code >> partially done
  • what about {{pageId}} ? and TYPO3_PAGE_ID in code ? >> partially replaced, id=<number> will still work in T3 10
#24

Updated by Marc Egger about 1 month ago

TODO NEXT: go through todos of thish ticket

#25

Updated by Marc Egger 30 days ago

TODO NEXT: habe die einfachen alias matches gefixed. jetzt megger/forkred testen was tut, was nicht

#26

Updated by Marc Egger 20 days ago

Gespraech mit Carsten:

Entscheidungen:
  • Definitiv Slug migration ab T3 v9. as _link funktioniert nur noch mit slugs ab v9.
todo (bereits in ticket uebertragen):
  • Release notes text aufsetzen.
    • "migration muss gemacht werden sonst funktioniert as _link nicht mehr"
  • $systemMessage feature in dev doku doukmentieren und bei variabel definition link dort hin
  • T3 v8 switches einbauen
  • prefix von report datei in exception angeben
  • qfq.json pfad angeben in exception
  • schauen ob ich in pre html tags zeilennummern anzeigen kann
    • ist moeglich mit CSS aber super aufwaendig. lohnt sich nicht.
  • was machen wir mit uploads page in forkred? > wahrscheinlich bei Simon UZH mob passiert
  • Hinweis geben, dass ein QFQ tt-content deaktiviert ist oder die page darauf. "tt-content is probably not in use"
  • andere QFQ installationen migrieren und testen
#27

Updated by Marc Egger 19 days ago

TODO NEXT: todos mit A machen

#28

Updated by Marc Egger 15 days ago

TODO NEXT: Remove Carsten fixes using BASE_DIR_FAKE. Now fix everything.

command to run phpunit:

www-data@webwork16:/var/www/html/megger/forkred/typo3conf/ext/qfq$ vendor/bin/phpunit --configuration phpunit.xml

Also available in: Atom PDF