Hi,

i would like to know about concepts of migration of a typo3 latin1
database to utf-8.
i helped myself by taking a dump and converting it with iconv but that
can not solve problems e.g. caused by serialized strings, where
stringlength might differ after conversion.

is there something known like a standard-procedure to manage such a
migration in typo3 context?

cheers
tom
_______________________________________________
TYPO3-english mailing list
TYPO3-english (AT) lists (DOT) netfielders.de
http://lists.netfielders.de/cgi-bin/.../typo3-english

Thomas Oppelt wrote:

> i would like to know about concepts of migration of a typo3 latin1
> database to utf-8.
> i helped myself by taking a dump and converting it with iconv but that
> can not solve problems e.g. caused by serialized strings, where
> stringlength might differ after conversion.
>
> is there something known like a standard-procedure to manage such a
> migration in typo3 context?

Google is your friend:

http://www.google.ca/search?q=typo3+utf-8

First hit!

Then:
http://wiki.typo3.org/index.php/UFT-...abase_to_UTF-8



Patrick
_______________________________________________
TYPO3-english mailing list
TYPO3-english (AT) lists (DOT) netfielders.de
http://lists.netfielders.de/cgi-bin/.../typo3-english

On Jan 11, 2008, at 3:10 PM, Patrick Gaumond wrote:

> Thomas Oppelt wrote:
>
>> i would like to know about concepts of migration of a typo3 latin1
>> database to utf-8.
>> i helped myself by taking a dump and converting it with iconv but
>> that
>> can not solve problems e.g. caused by serialized strings, where
>> stringlength might differ after conversion.
>>
>> is there something known like a standard-procedure to manage such a
>> migration in typo3 context?
>
> Google is your friend:
>
> http://www.google.ca/search?q=typo3+utf-8
>
> First hit!
>
> Then:
> http://wiki.typo3.org/index.php/UFT-...abase_to_UTF-8
>
>
>
> Patrick


one warning and is shown wrong on many links,
when you use a resent version of mysqldump, then you don't have to
encode your
dump with iconv or other tools since the dump is already in utf-8.

Ries





_______________________________________________
TYPO3-english mailing list
TYPO3-english (AT) lists (DOT) netfielders.de
http://lists.netfielders.de/cgi-bin/.../typo3-english

Hi Patrick,

thanks vor hints, but: is "recode" able to handle serialized strings
conversion rather then iconv? Sorry i am not that familiar with both
ways. Same question for me according to the wiki example where the dump
is not converted at all as i see, just inserted back with
--default-character-set=utf8. Is this enough and fits all potential
problems?
I can try this myself, but maybe you are so familiar with charset
conversion stuff that you can give me a simple "yes, that's all no
problem".

thx & regards
tom

Patrick Gaumond schrieb:
> Thomas Oppelt wrote:
>
>> i would like to know about concepts of migration of a typo3 latin1
>> database to utf-8.
>> i helped myself by taking a dump and converting it with iconv but that
>> can not solve problems e.g. caused by serialized strings, where
>> stringlength might differ after conversion.
>>
>> is there something known like a standard-procedure to manage such a
>> migration in typo3 context?
>
> Google is your friend:
>
> http://www.google.ca/search?q=typo3+utf-8
>
> First hit!
>
> Then:
> http://wiki.typo3.org/index.php/UFT-...abase_to_UTF-8
>
>
>
>
> Patrick
_______________________________________________
TYPO3-english mailing list
TYPO3-english (AT) lists (DOT) netfielders.de
http://lists.netfielders.de/cgi-bin/.../typo3-english

ries van Twisk schrieb:
>
> On Jan 11, 2008, at 3:10 PM, Patrick Gaumond wrote:
>
>> Thomas Oppelt wrote:
>>
>>> i would like to know about concepts of migration of a typo3 latin1
>>> database to utf-8.
>>> i helped myself by taking a dump and converting it with iconv but that
>>> can not solve problems e.g. caused by serialized strings, where
>>> stringlength might differ after conversion.
>>>
>>> is there something known like a standard-procedure to manage such a
>>> migration in typo3 context?
>>
>> Google is your friend:
>>
>> http://www.google.ca/search?q=typo3+utf-8
>>
>> First hit!
>>
>> Then:
>> http://wiki.typo3.org/index.php/UFT-...abase_to_UTF-8
>>
>>
>>
>>
>> Patrick
>
>
> one warning and is shown wrong on many links,
> when you use a resent version of mysqldump, then you don't have to
> encode your
> dump with iconv or other tools since the dump is already in utf-8.
>
> Ries

But does this prevent me from these cases where a s:4:"öäüß"; in latin1
is s:8:"öäüß"; in utf-8.
So if you try to unserialize such a string after conversion u get just
false, or not?



_______________________________________________
TYPO3-english mailing list
TYPO3-english (AT) lists (DOT) netfielders.de
http://lists.netfielders.de/cgi-bin/.../typo3-english

Thomas Oppelt wrote: on 12.01.2008 11:43:

>>> http://www.google.ca/search?q=typo3+utf-8
>>>
>>> First hit!
>>>
>>> Then:
>>> http://wiki.typo3.org/index.php/UFT-...abase_to_UTF-8

>> one warning and is shown wrong on many links,
>> when you use a resent version of mysqldump, then you don't have to
>> encode your
>> dump with iconv or other tools since the dump is already in utf-8.
>>
>> Ries
>
> But does this prevent me from these cases where a s:4:"öäüß"; in latin1
> is s:8:"öäüß"; in utf-8.
> So if you try to unserialize such a string after conversion u get just
> false, or not?

I guess you will have to "script" that converter on your own, because no
"automatic" converter will work with PHP-serialized array. I haven't
really found the need to do that, in most cases the serialized arrays
are either some "minor" configuration that can be adapted by hand after
converting everything else, or some cached stuff that we can clean up
afterwards anyway. Isn't that the case?

Cheers,
Ernesto
_______________________________________________
TYPO3-english mailing list
TYPO3-english (AT) lists (DOT) netfielders.de
http://lists.netfielders.de/cgi-bin/.../typo3-english

Ernesto Baschny [cron IT] schrieb:
> Thomas Oppelt wrote: on 12.01.2008 11:43:
>
>>>> http://www.google.ca/search?q=typo3+utf-8
>>>>
>>>> First hit!
>>>>
>>>> Then:
>>>> http://wiki.typo3.org/index.php/UFT-...abase_to_UTF-8
>
>
>>> one warning and is shown wrong on many links,
>>> when you use a resent version of mysqldump, then you don't have to
>>> encode your
>>> dump with iconv or other tools since the dump is already in utf-8.
>>>
>>> Ries
>>
>> But does this prevent me from these cases where a s:4:"öäüß"; in
>> latin1 is s:8:"öäüß"; in utf-8.
>> So if you try to unserialize such a string after conversion u get just
>> false, or not?
>
> I guess you will have to "script" that converter on your own, because no
> "automatic" converter will work with PHP-serialized array. I haven't
> really found the need to do that, in most cases the serialized arrays
> are either some "minor" configuration that can be adapted by hand after
> converting everything else, or some cached stuff that we can clean up
> afterwards anyway. Isn't that the case?
>
> Cheers,
> Ernesto

i HOPE this is the case, i don't know if typo3 just stores temp data
like caching data as serialized array/strings...what be nice if someone
could say "no prob u can be unconcerned about that".
_______________________________________________
TYPO3-english mailing list
TYPO3-english (AT) lists (DOT) netfielders.de
http://lists.netfielders.de/cgi-bin/.../typo3-english

_______________________________________________
TYPO3-english mailing list
TYPO3-english (AT) lists (DOT) netfielders.de
http://lists.netfielders.de/cgi-bin/.../typo3-english

On Jan 15, 2008, at 11:02 AM, Steffen Kamper wrote:

>
>>> <-....snip>
>>
>> I guess you will have to "script" that converter on your own,
>> because no
>> "automatic" converter will work with PHP-serialized array. I haven't
>> really found the need to do that, in most cases the serialized
>> arrays are
>> either some "minor" configuration that can be adapted by hand after
>> converting everything else, or some cached stuff that we can clean up
>> afterwards anyway. Isn't that the case?
>>
>> Cheers,
>> Ernesto
>
> i don't think that is needed. As a dump is a normal textfile there
> is no
> difference between text and serialized array, conversion will be
> done for
> complete text.
>
> vg Steffen
>
>

Hey Steffen

the problem is the serialized arrays.
What happens is that if an character get's changed to a two byte
notation instead
of a one byte notation the length parameter of the serialized array is
incorrect.

The length parameters counts bytes, not characters.

If that happens then unserialize will fail and return false (or was it
NULL? )

Ries



--
Ries van Twisk
Freelance TYPO3 Developer
email: ries (AT) vantwisk (DOT) nl
web: http://www.rvantwisk.nl/
skype: callto://r.vantwisk
Phone: + 1 810-476-4193







_______________________________________________
TYPO3-english mailing list
TYPO3-english (AT) lists (DOT) netfielders.de
http://lists.netfielders.de/cgi-bin/.../typo3-english

"ries van Twisk" <typo3 (AT) rvt (DOT) dds.nl> schrieb im Newsbeitrag
news:mailman.3823.1200430939.5304.typo3-english (AT) lists (DOT) netfielders.de...
>
> On Jan 15, 2008, at 11:02 AM, Steffen Kamper wrote:
>
>>
>>>> <-....snip>
>>>
>>> I guess you will have to "script" that converter on your own, because
>>> no
>>> "automatic" converter will work with PHP-serialized array. I haven't
>>> really found the need to do that, in most cases the serialized arrays
>>> are
>>> either some "minor" configuration that can be adapted by hand after
>>> converting everything else, or some cached stuff that we can clean up
>>> afterwards anyway. Isn't that the case?
>>>
>>> Cheers,
>>> Ernesto
>>
>> i don't think that is needed. As a dump is a normal textfile there is no
>> difference between text and serialized array, conversion will be done
>> for
>> complete text.
>>
>> vg Steffen
>>
>>
>
> Hey Steffen
>
> the problem is the serialized arrays.
> What happens is that if an character get's changed to a two byte notation
> instead
> of a one byte notation the length parameter of the serialized array is
> incorrect.
>
> The length parameters counts bytes, not characters.
>
> If that happens then unserialize will fail and return false (or was it
> NULL? )
>
> Ries

Hi Ries,

indeed i didn't thought about that.
But i remember that i took a dump file, changed collation and charset
manually and restored db, and result was ok.

What's about the extension convert2utf8 - does it works? maybe if not it
could be improved. Download count is nearly 3000, so it could help.

vg Steffen


_______________________________________________
TYPO3-english mailing list
TYPO3-english (AT) lists (DOT) netfielders.de
http://lists.netfielders.de/cgi-bin/.../typo3-english