[patch] unicode problem with DJabberd::RosterStorage::SQLite

[Brad Fitzpatrick]

That's a hacky fix without an explanation.

Explain to me which layers/modules want SvUTF8 strings, which don't, and
where the malformed conversion is happening, and then let's do a small,
surgical fix.  Not a massive "use bytes" hammer at the top that might fix it
but might break something else that was expecting SvUTF8 scalars.  (perhaps
libxml in the XML stream to users, for instance, or libxml parsing
itself....)

On Fri, Jan 23, 2009 at 12:48 AM, Aleksey Luzin <subscr at p34.net> wrote:

> Hello,
>
> Is this issue still exists ?
> I have fixed russian nicknames corruptions by adding string
> use bytes;
> in module IQ.pm
>
> On Friday 26 September 2008 19:25:49 Michael Scherer wrote:
> > Le samedi 22 septembre 2007 à 20:56 +0000, Brad Fitzpatrick a écrit :
> > > A perl program has to use Perl's UTF8 scalars everywhere, or nowhere,
> > > otherwise introducing them (as this patch does) starts the contagious
> > > spread of them... whenever one touches a non-UTF8 scalar, perl starts
> > > converting the one that's touched.
> > >
> > > I imagine we do use it, though, as XML libraries tend to use it?
> > >
> > > So this _should_ be safe and the correct fix (I've seen the same bug,
> > > btw), but I want somebody to double-check we are using Perl's UTF-8 for
> > > djabberd...
> > >
> > > Can somebody look at Devel::Peek on a scalar (say a username) when a
> > > non-ASCII username comes in, and see if the SvUTF8 flag is on?
> >
> > Since it seems no one did it, I decided to take a closer look at this
> > request, and it seems the UTF8 flags is already set with or without my
> > patch :
> >
> >
> > without it :
> >
> > SV = RV(0x8c64024) at 0x8c64018
> >   REFCNT = 1
> >   FLAGS = (PADMY,ROK)
> >   RV = 0x8e12820
> >   SV = PVAV(0x8e61cb8) at 0x8e12820
> >     REFCNT = 6
> >     FLAGS = (OBJECT,OVERLOAD)
> >     STASH = 0x86d7d28 "DJabberd::JID"
> >     ARRAY = 0x8eecd18
> >     FILL = 5
> >     MAX = 11
> >     ARYLEN = 0x0
> >     FLAGS = (REAL)
> >     Elt No. 0
> >     SV = PVMG(0x8da44dc) at 0x8e29b40
> >       REFCNT = 1
> >       FLAGS = (POK,pPOK,UTF8)
> >       IV = 0
> >       NV = 0
> >       PV = 0x8d9b530 "phoebe"\0 [UTF8 "phoebe"]
> >       CUR = 6
> >       LEN = 8
> >     Elt No. 1
> >     SV = PVMG(0x8da4300) at 0x8f04940
> >       REFCNT = 1
> >       FLAGS = (POK,pPOK,UTF8)
> >       IV = 0
> >       NV = 0
> >       PV = 0x8dfb378 "example.org"\0 [UTF8 "example.org"]
> >       CUR = 11
> >       LEN = 12
> >     Elt No. 2
> >     SV = PVMG(0x8da4290) at 0x8e29770
> >       REFCNT = 1
> >       FLAGS = (POK,pPOK,UTF8)
> >       IV = 0
> >       NV = 0
> >       PV = 0x8da1410 "Sisay"\0 [UTF8 "Sisay"]
> >       CUR = 5
> >       LEN = 8
> >     Elt No. 3
> >     SV = PV(0x8db6030) at 0x8e38a80
> >       REFCNT = 1
> >       FLAGS = (POK,pPOK,UTF8)
> >       PV = 0x8eecd50 "phoebe at example.org/Sisay"\0 [UTF8
> > "phoebe at example.org/Sisay"]
> >       CUR = 24
> >       LEN = 28
> >
> >
> > and with the patch
> >
> > SV = RV(0x8c642cc) at 0x8c642c0
> >   REFCNT = 1
> >   FLAGS = (PADMY,ROK)
> >   RV = 0x8f83560
> >   SV = PVAV(0x8ef2df8) at 0x8f83560
> >     REFCNT = 6
> >     FLAGS = (OBJECT,OVERLOAD)
> >     STASH = 0x86d7d28   "DJabberd::JID"
> >     ARRAY = 0x8ef5eb0
> >     FILL = 5
> >     MAX = 11
> >     ARYLEN = 0x0
> >     FLAGS = (REAL)
> >     Elt No. 0
> >     SV = PVMG(0x8e37b08) at 0x8e75188
> >       REFCNT = 1
> >       FLAGS = (POK,pPOK,UTF8)
> >       IV = 0
> >       NV = 0
> >       PV = 0x8d92f88 "phoebe"\0 [UTF8 "phoebe"]
> >       CUR = 6
> >       LEN = 8
> >
> >     Elt No. 1
> >     SV = PVMG(0x8eb7cfc) at 0x8f10228
> >       REFCNT = 1
> >       FLAGS = (POK,pPOK,UTF8)
> >       IV = 0
> >       NV = 0
> >       PV = 0x8ea5568 "example.org"\0 [UTF8 "ephaone.org"]
> >       CUR = 11
> >       LEN = 12
> >     Elt No. 2
> >     SV = PVMG(0x8e37aec) at 0x8eed410
> >       REFCNT = 1
> >       FLAGS = (POK,pPOK,UTF8)
> >       IV = 0
> >       NV = 0
> >       PV = 0x8e58dd8 "Sisay"\0 [UTF8 "Sisay"]
> >       CUR = 5
> >       LEN = 8
> >     Elt No. 3
> >     SV = PV(0x8de9dc0) at 0x8f02360
> >       REFCNT = 1
> >       FLAGS = (POK,pPOK,UTF8)
> >       PV = 0x8f08a78 "phoebe at example.org/Sisay"\0 [UTF8
> > "phoebe at ephaone.org/Sisay"]
> >       CUR = 24
> >       LEN = 28
> >
> >
> >
> > So i guess that, if I add that i have been running the server since one
> > year without problem, that the patch is sae enough to be pushed
> > ( the one i posted on
> > http://lists.danga.com/pipermail/djabberd/2007-September/000401.html )
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/djabberd/attachments/20090123/c4182298/attachment.html