1 21:07:25 - GuKKDevel: did you get my messaage on bug-1305?
   2 21:08:26 - ted: Yes.
   3 21:08:41 - GuKKDevel: from all the comments I think we could try to include the SH256 fingerprints and the SHA1
   4 21:09:07 - ted: Yes, I also got the impression
   5 21:09:26 - ted: But we don't change the layout otherwise
   6 21:10:10 - GuKKDevel: no, only a third line in he header
   7 21:10:21 - GuKKDevel: /he/the
   8 21:10:33 - ted: Let me have a look once more...
   9 21:10:51 - GuKKDevel: and with the linebreak I have to try
  10 21:11:56 - ted: OK, three lines should be OK and fit even on Letter paper.
  11 21:13:08 - ted: Don't invest too much time into the linebreaks. I really guess that this was so from the "beginning of time"... One of the reasons why I always used the english form.
  12 21:13:15 - GuKKDevel: the boxes are calculated to fit in the paper at that position, above is more space that can be used
  13 21:13:37 - ted: Ahh...
  14 21:13:54 - ted: So the boxes are positioned at absolute coordinates?
  15 21:13:59 - GuKKDevel: yes
  16 21:14:02 - ted: This is good.
  17 21:15:00 - ted: Do you already have an idea why switching the language does not work smoothly?
  18 21:16:05 - GuKKDevel: could be the same fact as it is in CATS when you go back and reach with this the 100%
  19 21:17:03 - GuKKDevel: it seems, that the class L10n is not called a second time
  20 21:17:21 - GuKKDevel: after changing the language
  21 21:18:22 - ted: If there is an easy fix (adding one call after changing the language), do it. If not, we'll leave it for now.
  22 21:18:32 - GuKKDevel: another point, we have to look if the php/mysql-environment is correctly adjusted with utf-8
  23 21:19:29 - GuKKDevel: I agree to leave it. it should be seldom to change the language to print the CAP-forms
  24 21:19:30 - ted: Hmm, is UTF-8 already used?
  25 21:21:57 - ted: It's a bit hard to tell with the manual SQL interface, but to me it looks like the german umlauts are stored in a single byte...
  26 21:22:08 - GuKKDevel: yes this was what went wrong when we changed the PHP/OS version; for some month we got problems with diacritics https://bugs.cacert.org/view.php?id=1441
  27 21:23:06 * ted sighs deeply
  28 21:23:33 - GuKKDevel: wytze repaired it
  29 21:23:46 - ted: At the moment I'd prefer to store only 7 bit ASCII and use HTML encoding for everything else
  30 21:24:41 - ted: This should work regardless of any codepage setting of the database or OS
  31 21:24:43 - GuKKDevel: I'm not so deep in that things, with configurations and so
  32 21:25:43 - ted: I also don't know exactly how the database and PHP handle non-ASCII
  33 21:26:25 - GuKKDevel: ahh I found in the bug that wytze configured iso-8859-1
  34 21:28:20 - ted: Yes, it looks like, at least on the testserver database, "ö" is stored in iso-8859-1.
  35 21:29:03 - ted: But, this has nothing to do with the new certificates, that's part of the migration job, correct?
  36 21:29:19 - GuKKDevel: I think so
  37 21:30:22 - ted: OK, so, firstly you should give me a CAP form with SHA1 fingerprints.
  38 21:30:53 - GuKKDevel: will do it in the next days
  39 21:30:54 - bdmc [bdmc@user-0c8huj1.cable.mindspring.com] hat den Raum betreten.
  40 21:31:20 - bdmc: Sorry to be late.
  41 21:31:33 - GuKKDevel: hi brian
  42 21:31:54 - bdmc: 'Morning, Karl-Heinz
  43 21:31:54 - ted: Then we should try to get non-ASCII data stored in the database as HTML encoded. I guess that's the best way to migrate to the final goal of UTF-8
  44 21:31:59 - ted: Hi Brian!
  45 21:32:25 - bdmc: 'Evening, Ted.
  46 21:33:00 - ted: I'm afraid that we'll have to run some database scripts on the production DB to fix those encoding issues... :-(
  47 21:33:24 - GuKKDevel: I'm not eager to store non-ASCII.
  48 21:34:05 - bdmc: Another option, and it may be no better, is to migrate to a "new" database, with the correct encoding.
  49 21:34:12 - GuKKDevel: I just thought about the error as in bug-1441
  50 21:34:23 - ted: Hmm, let me try something on the testserver...
  51 21:35:23 - GuKKDevel: for fixing these encoding issues we are waiting for an arbitrator to give us allowance
  52 21:36:04 - ted: Just want to try how something like Walesa is stored. That's non-Latin-1
  53 21:38:00 - ted: $&%/
  54 21:38:16 - ted: Mail address verification prohibits registration of new accounts...
  55 21:38:50 - bdmc: Does the mail system allow non-ASCII?
  56 21:38:57 - bdmc: For addresses?
  57 21:39:24 - ted: The problems are not the mail addresses
  58 21:39:30 - bdmc: OK
  59 21:39:35 - ted: those should all be ASCII#
  60 21:39:46 - bdmc: Names and things like that, correct?
  61 21:40:30 - ted: Yes. Most of all names, and to a smaller amount comments, for example on certificates.
  62 21:41:15 - bdmc: Do we allow that in our advertised rules?  ( non-ASCII? )
  63 21:41:47 - ted: At least on the testserver I'm registered as "Fröhlich"
  64 21:42:39 - bdmc: Is that explicit in the rules, or just because it is "not forbidden?"   It's been too long since I studied those rules.
  65 21:43:26 - bdmc: PHP's "natural language" is utf-8.
  66 21:43:36 - ted: The rules encourage using of "correct" characters, but allow common substitutions like ö -> oe (in german)
  67 21:44:33 - bdmc: So, as long as we can Input and Output a character, we should also store it. ( must )
  68 21:45:15 - bdmc: How is the Arbitration situation coming?
  69 21:46:18 - GuKKDevel: bdmc: are you in contact with the treasurer?
  70 21:46:29 - ted: So, I just checked, at least in comments non Latin-1-characters are stored in the database as HTML encoded. Which is OK.
  71 21:46:48 - bdmc: No.  I saw your message, and wondered whether you had got a response.  Have you tried Etienne?
  72 21:47:14 - GuKKDevel: bdmc: no not yet
  73 21:47:52 - bdmc: GuKKDevel: I sent him a message about a week ago, you might have seen it, and have had no response either.
  74 21:48:16 - ted: So "Walesa" is stored as "Wałęsa". But when displaying the data the ampersand is itself encoded to & and so the comment is displayed garbled, exactly as it is in the database.
  75 21:48:49 - bdmc: ted: OK, so does this get us --- so input and output are not commutative?
  76 21:49:15 - ted: Looks like.
  77 21:49:39 - bdmc: That sounds like an error in htmlentities() or one of the other translation functions.  ( mis-use of that pair of functions )
  78 21:49:49 - ted: But IMHO this is no big issue, because it can be fixed by simple code changes, and the data in the database is correct.
  79 21:50:13 - bdmc: As long as it is going IN to the database correctly.
  80 21:50:23 - ted: Exactly.
  81 21:50:51 - bdmc: You should be able to use the MySQL command line to confirm that.
  82 21:51:15 - bdmc: Which you already did.
  83 21:51:23 - ted: Probably the other characters which are part of Latin-1, as ÄÖÜ, will be more trouble when trying to migrate to UTF-8 consistently.
  84 21:52:00 - bdmc: Do they not have appropriate utf-8 equivalents?
  85 21:53:14 - GuKKDevel: we have a function utf8_to_ascii
  86 21:53:32 - ted: That's not the problem, but they are stored in the database as one single byte.
  87 21:53:52 - bdmc: Sounds like we might need "latin1_to_utf8()".
  88 21:54:02 - GuKKDevel: which creates a database for transforming UTF-8 into ASCII
  89 21:54:05 - ted: So those have to be changed in the database bevore we can switch the database "codepage" to UTF-8
  90 21:54:24 - ted: Excactly. :-)
  91 21:54:47 - bdmc: I would rather copy to new tables, instead of trying to "convert in place."  ( new tables in a new database )
  92 21:54:52 - ted: Or "latin1_to_HTML()"
  93 21:55:19 - bdmc: That way we don't damage the original database.  ( don't have a chance to do so )
  94 21:55:48 - ted: Yes, probably this will be best, at least for important tables like USERS.
  95 21:56:05 - ted: I guess we can live with a few garbled comments.
  96 21:57:02 - bdmc: I haven't tried to do "insert into X ( select * from Y )" where X and Y are in different databases with different encodings.  ( different databases, just not different encoding )
  97 21:57:32 - ted: That's the reason why I'd prefer HTML encoding as an intermediate.
  98 21:58:14 - bdmc: But if my statement works, then we let MySQL do all of the heavy lifting, and don't try to code around it.
  99 21:58:35 - ted: You can store HTML encoded data (which is 7 bit ASCII) in the table, then change the database encoding to UTF-8
 100 21:59:21 - bdmc: Understood, and yes, that is a good idea, if you don't want "native encoding."   The HTML will always be HTML, no matter what the database encoding is.
 101 21:59:35 - ted: Yes.
 102 21:59:53 - ted: And if we want to get it as native-UTF-8 it's easy to do.
 103 22:00:04 - ted: Later...
 104 22:00:30 - bdmc: If we want to convert the Latin1 characters the UTF-8 characters --- OK.  If you want to delay things.
 105 22:00:52 - bdmc: ( the UTF ==> to UTF )
 106 22:01:35 - ted: I strongly assume that we already have HTML encoded characters in the database, since al those eastern Europe people are not covered with Latin-1
 107 22:02:21 - ted: Czech, Polish, Turkish, ...
 108 22:02:53 - bdmc: True.  So the system was already doing HTML encoding, "naturally?"
 109 22:03:19 - ted: Yes, looks like.
 110 22:03:36 - ted: What cannot be encoded in Latin-1 gets stored as HTML
 111 22:04:06 - bdmc: OK.  Hmmm.  How would it tell?  Test the PHP character for a certain range of values?
 112 22:04:37 - bdmc: I don't think that MySQL would do this on its own.
 113 22:04:51 - ted: I'm not sure at which level this is handled. May be already in the browser, or may be PHP.
 114 22:05:10 - ted: MySQL is most probably not doing anything.
 115 22:05:11 - bdmc: Interesting.  More research.
 116 22:05:23 - GuKKDevel: I noticed the use of htmlentities
 117 22:06:09 - GuKKDevel:  htmlentities()
 118 22:06:31 - bdmc: Yes, that is the function, and html_entity_decode(), if it is done correctly.
 119 22:06:37 - ted: I know that, if you set Latin-1 as charset in the HTML header, already the browser sends Non-Latin-1 as HTML entities.
 120 22:06:55 - ted: But it looks like (at least the testserver) sets utf-8 in the HTML header
 121 22:07:08 - bdmc: ted: Ahhh.  Thank you.  So it is happening outside our code.
 122 22:08:17 - bdmc: But that means that if we try to encode things, in the same way, we might ( probably ) get conflicts, like that mis-transalated "&".
 123 22:09:25 - bdmc: As usual, we need to CAREFULLY handle Input and Output.
 124 22:10:11 - ted: The production server seems to use "windows-1252" in the HTTP-header, the HTML header does not specify a character set. So indeed I guess that in the productive system the browser already does the work.
 125 22:11:30 - bdmc: Good old "windows-1252."
 126 22:11:50 - bdmc: Not exactly Linux.
 127 22:12:18 - ted: As usual in the area of character encoding, things are quite messy... :-\
 128 22:14:31 - ted: Now for something completely different...
 129 22:14:57 - GuKKDevel:  another question; I trying to document the current system. Now I have a problem with understanding the flow logic.
 130 22:15:09 - ted: I guess that it would make sense to disable the mail address checking when registering on the testserver.
 131 22:15:34 - ted: OK,. GuKK, what's your problem?
 132 22:15:42 - GuKKDevel: if I enter www.cacert.org I will get to index.php
 133 22:16:05 - GuKKDevel: in index php there is an include to two other files
 134 22:16:52 - bdmc: general.php, in particular.   It comes in from the .htaccess file, as well.
 135 22:17:10 - GuKKDevel: then there is a function call loadem, tht function is in includes/general.php but I can't find the place where this file is included
 136 22:17:37 - GuKKDevel: ahhhh thanks bdmc
 137 22:17:39 - ted: Yes. Me neither.
 138 22:17:40 - bdmc: includes/general.php, which calls includes/lib/general.php,
 139 22:18:15 - ted: .htaccess is the culprit???
 140 22:18:19 - bdmc: that .htaccess got me too.   I don't like it.  I would like to find a different way to do that.
 141 22:19:12 - ted: Which .htaccess?
 142 22:19:13 - bdmc: It probably wouldn't be difficult.  We would just need to agree.
 143 22:19:27 - bdmc: The one in "root".
 144 22:19:34 - bdmc: ( above www )
 145 22:20:14 - /home/cacert ist kein unterstützter Befehl. Geben Sie /help ein, um eine Liste verfügbarer Befehle zu sehen.
 146 22:20:19 - ted:  /home/cacert or /home/cacert/www (there is none in both on the testserver)
 147 22:20:32 - bdmc: Sorry.  It is IN www.
 148 22:20:58 - bdmc: If you don't have ".htaccess" in www, your installation is incomplete.
 149 22:21:23 - ted: Yes, you're right. One bonus point for you! :-)
 150 22:21:48 - GuKKDevel: is in /home/cacert/www/www
 151 22:21:50 - bdmc: Awww.  You're too kind.
 152 22:22:09 - bdmc: I don't know how your directories are laid out.
 153 22:22:21 - ted: OK I hate this trick. Let's get rid of it in the course of the migration to PHP 7
 154 22:22:40 - GuKKDevel: the first www is the mirror link to the git
 155 22:23:51 - bdmc: There are two different things, like this, in the .htaccess file.  There is also something that I would expect to find in the Apache configuration, pointing at the CPS.
 156 22:23:58 - ted: The like "php_value auto_prepend_file /www/includes/general.php" looks very much like the culprit.
 157 22:24:13 - bdmc: That's the one.
 158 22:24:34 - ted: The redirect is also not really nice.
 159 22:24:41 - bdmc: Agreed.
 160 22:25:10 - bdmc: That should ( at least in my opinion ) be in the Virtual Host configuration.
 161 22:25:37 - bdmc: ( or general configuration )
 162 22:25:44 - ted: Or do it in cps.php irself.
 163 22:26:06 - ted: Why hide it in configuration when it's content?
 164 22:26:08 - bdmc: What this does is "remove" cps.php.
 165 22:26:20 - ted: Yes, of course.
 166 22:26:57 - GuKKDevel: this is a way to undergo the necessity to have changes checked
 167 22:27:01 - bdmc: But, I agree.  Why "rename" the HTML file with this redirect?  Just call the HTML file cps.php, and be done with it.
 168 22:27:54 - bdmc: GuKKDevel: I don't understand.  Don't all files need to be checked at some point?
 169 22:27:54 - GuKKDevel: the file cps.php is under control of software and critical, the other file is not
 170 22:28:16 - bdmc: Ahhh.  OK.  Hmmm.
 171 22:28:24 - ted: There may be good reasons why the HTML file is not renamed. I'm afraid that GuKK is right.
 172 22:28:42 - bdmc: So the files in "policy" are not "critical?"
 173 22:28:56 - GuKKDevel: there can policy-group do the changes and doesn't have to wait for a software
 174 22:29:26 - bdmc: OK.  So create cps.php and have it "include" the "policy" file.
 175 22:29:53 - bdmc: That way, any changes in Policy are immediately refected in cps.php, but no code changes.
 176 22:30:04 - GuKKDevel: but in that case policy is still out of control of software
 177 22:30:29 - bdmc: I thought that that was the goal.
 178 22:30:48 - GuKKDevel: I don't know what the goal was
 179 22:31:14 - ted: Maybe the reason is just that CertificationPracticeStatement.html is edited in the version management under this name. And it IS a better name than cps.php.
 180 22:32:07 - bdmc: Which is reasonable.  is "cps.php" called internally?
 181 22:32:50 - ted: I guess (don't know for sure) that cps.php is considerably oder.
 182 22:32:58 - ted: oder -> older
 183 22:33:41 - ted: CertificationPracticeStatement.html was edited by policy group, only "a few years ago"
 184 22:34:15 - bdmc: Hmmm.  I just searched the code, and I don't find a reference to cps.php.
 185 22:34:58 - ted: It may be in the WiKi or on external websites
 186 22:35:24 - ted: No way to be sure that there are none remaining!
 187 22:35:40 - ted: (others than looking for 404 errors)
 188 22:36:01 - bdmc: I went up a level, and found a couple of references.  More to investigate.  ( later )
 189 22:37:24 - GuKKDevel: cps.php is called in pages/account/3.php
 190 22:37:42 - ted: So, let's add a dummy cps.php which just includes the HTML file, and remove the .htaccess magic.
 191 22:38:05 - ted: No big deal
 192 22:38:08 - bdmc: pages/account/3.php, if anybody is interested.
 193 22:38:36 - bdmc: GuKKDevel: you beat me.  B-)
 194 22:38:46 * GuKKDevel grins
 195 22:40:57 - GuKKDevel: but includes/general.php should stay, because we had to include it in all other files prophylactical
 196 22:41:22 * ted sighs deeply.
 197 22:41:57 - bdmc: True, but we can do that in a different way than .htaccess.
 198 22:43:43 - bdmc: We only need to do this in the "start" programs.  The programs that are included in those "start" programs don't need general.php.
 199 22:44:26 - ted: Yes, so probably only the files in www/www, without the subdirectories.
 200 22:45:03 - ted: And maybe api and cats (though I doubt that the cats api needs general.php)
 201 22:45:17 - bdmc: Should be fewer than that.  Perhaps only index.php, but I would want to check that.
 202 22:46:08 - bdmc: ted: You're probably right.
 203 22:46:32 - GuKKDevel: cats needs sanitize from general.php
 204 22:49:24 - GuKKDevel: wrong has its own function defined
 205 22:49:33 - ted: GuKK: It only needs sanitize_string, which is implemented in cats_import.php
 206 22:50:05 - ted: But anyway, I'd really prefer to explicitly include that file.
 207 22:50:17 - ted: (at least when it is needed)
 208 22:50:25 - GuKKDevel: so do I
 209 22:50:45 - bdmc: Agreed
 210 22:51:14 - ted: It does quite many things. I now dimly remember that I also found this out some time ago...
 211 22:51:14 - GuKKDevel: and as I'm documenting, I could do the include to each file that needs a function from general
 212 22:51:39 - ted: And since I forgot it agains it is important to be documented! :-)
 213 22:52:04 - bdmc: Documentation -- who needs documentation?   B-)
 214 22:52:27 * ted giggles inanely.
 215 22:52:45 - bdmc: Combination of general and lib/general.
 216 22:52:53 - ted: Can it be that there are problems when general.php id included twice?
 217 22:52:57 - bdmc: And the other files that they include.
 218 22:53:19 - bdmc: require_once() would take care of that.
 219 22:53:55 - ted: Yes, but it would explain the problems I had when playing around with your bug branch, Brian.
 220 22:55:28 - bdmc: Hmm.  Good point.  Yes, I had included it manually, during my own testing, and it somehow got included in the committed files.
 221 22:55:35 - ted: Nope, there was a "require_once"
 222 22:56:29 - bdmc: require_once() makes sure that an include file is only included once.
 223 22:57:12 - GuKKDevel: see I correct, include also as require_once includes the file at the place where the include/require-statement is?
 224 22:57:17 - ted: Whatever... maybe "auto_prepend_file" is not counted as an include...
 225 22:57:51 - bdmc: It would seem to me that that would do it whether you wanted it to or not.
 226 22:57:58 - GuKKDevel: and if there is a logic in that file it is executed?
 227 22:58:28 - ted: Like it is in general.php
 228 22:59:18 - bdmc: GuKKDevel: Yes, includes and requires happen at the point in the source file where they occur.  Also yes, if there is executable code, not just functions, that code is executed at that point.
 229 22:59:37 - GuKKDevel: thanks
 230 23:00:28 - ted: So, this is the reason why it is needed in every top-level file. There's some setup code in there.
 231 23:01:07 - bdmc: True.  But including it at the top of index.php would take care of this issue.
 232 23:02:04 - GuKKDevel: so as there are only functions defined in includes/general.php and includes/lib/general.php there should be no problem to change the includes to require-once
 233 23:02:30 - bdmc: yes
 234 23:03:02 - GuKKDevel: and add a require_once for safety if a module from one of these files is used
 235 23:03:06 - ted: Include/general.php indeed does execute code!
 236 23:03:53 - ted: The first 100 lines of it are ouside of any function definition
 237 23:04:06 - GuKKDevel: I see
 238 23:04:42 - ted: It includes a whole lot of things...
 239 23:04:57 - GuKKDevel: so should we leave the auto-prepend statement and document it only in codedocs?
 240 23:05:00 - ted: and set session variables
 241 23:05:42 - ted: I really don't know what to do...
 242 23:06:41 - GuKKDevel: all things done there would go to the new class brian developes
 243 23:06:50 - bdmc: ted: That's all right.  As long as that "setup" code is only executed once, at the beginning of execution, then it is done correctly.  ( UNLESS that code needs to re-execute for some bizzarre reason. )
 244 23:07:15 - GuKKDevel: so why not wait until we can install this class?
 245 23:07:32 - ted: Maybe that's the trick...
 246 23:07:47 - bdmc: At the moment that class is for Configuration, but I suppose that we can do setup there, too.
 247 23:08:20 - GuKKDevel: or may be another class only for setup
 248 23:08:48 - bdmc: Sorry.  Yes, that was what I should have said.
 249 23:09:45 - ted: So I think, postponed for now?
 250 23:09:53 - GuKKDevel: postponed
 251 23:09:54 - ted: Just do the docu
 252 23:10:03 - bdmc: OK
 253 23:10:09 - GuKKDevel: OK
 254 23:10:56 - ted: ANyone still wants to talk about testserver-setup? I'm getting a bit tired..
 255 23:11:00 - GuKKDevel: I will do he docu for .htaccess as soon as possible
 256 23:11:28 - bdmc: ted: I'm good for about six hours.   B-)
 257 23:11:47 - bdmc: But I am quite willing to close.
 258 23:11:56 - GuKKDevel: so am I
 259 23:12:23 - ted: I guess it won't make sense to schedule another meeting this year.
 260 23:12:53 - bdmc: Up to you two, and others.  I'm not going anywhere.
 261 23:13:38 - GuKKDevel: lets try 2018-12-21
 262 23:13:55 - ted: How about next meeting at Jan 04? Or better Jan 11? Just in case we can have informal meetings inbetween. :-)
 263 23:14:02 - bdmc: Fine with me, either way.
 264 23:14:12 - ted: OK, then we'll tra Dec 21.
 265 23:14:35 * ted thinks carefully.
 266 23:15:14 - GuKKDevel: OK and  schedule the next for 04.1.19 in advance
 267 23:15:23 - ted: I won't be there Dec 21. But that should not disturb you.
 268 23:15:44 - GuKKDevel: if we do not meet, merry chrismas and a happy new year
 269 23:16:15 - bdmc: I can put them both in my calendar.
 270 23:16:31 - GuKKDevel: but we communicate on devel list
 271 23:16:38 - bdmc: We can definitely do that.
 272 23:16:41 - ted: And have a nice time sweating at the christmas tree, down under!
 273 23:17:00 - ted: I'll try to do some minutes. See you next year!
 274 23:17:33 - GuKKDevel: so good bye for now
 275 23:17:37 - bdmc: Have a very Happy Christmas, all!
 276 23:17:58 - ted: And to all of you!
 277 23:18:12 - GuKKDevel: same from me