21:07:25 - GuKKDevel: did you get my messaage on bug-1305?
21:08:26 - ted: Yes.
21:08:41 - GuKKDevel: from all the comments I think we could try to include the SH256 fingerprints and the SHA1
21:09:07 - ted: Yes, I also got the impression
21:09:26 - ted: But we don't change the layout otherwise
21:10:10 - GuKKDevel: no, only a third line in he header
21:10:21 - GuKKDevel: /he/the
21:10:33 - ted: Let me have a look once more...
21:10:51 - GuKKDevel: and with the linebreak I have to try
21:11:56 - ted: OK, three lines should be OK and fit even on Letter paper.
21:13:08 - ted: Don't invest too much time into the linebreaks. I really guess that this was so from the "beginning of time"... One of the reasons why I always used the english form.
21:13:15 - GuKKDevel: the boxes are calculated to fit in the paper at that position, above is more space that can be used
21:13:37 - ted: Ahh...
21:13:54 - ted: So the boxes are positioned at absolute coordinates?
21:13:59 - GuKKDevel: yes
21:14:02 - ted: This is good.
21:15:00 - ted: Do you already have an idea why switching the language does not work smoothly?
21:16:05 - GuKKDevel: could be the same fact as it is in CATS when you go back and reach with this the 100%
21:17:03 - GuKKDevel: it seems, that the class L10n is not called a second time
21:17:21 - GuKKDevel: after changing the language
21:18:22 - ted: If there is an easy fix (adding one call after changing the language), do it. If not, we'll leave it for now.
21:18:32 - GuKKDevel: another point, we have to look if the php/mysql-environment is correctly adjusted with utf-8
21:19:29 - GuKKDevel: I agree to leave it. it should be seldom to change the language to print the CAP-forms
21:19:30 - ted: Hmm, is UTF-8 already used?
21:21:57 - ted: It's a bit hard to tell with the manual SQL interface, but to me it looks like the german umlauts are stored in a single byte...
21:22:08 - GuKKDevel: yes this was what went wrong when we changed the PHP/OS version; for some month we got problems with diacritics https://bugs.cacert.org/view.php?id=1441
21:23:06 * ted sighs deeply
21:23:33 - GuKKDevel: wytze repaired it
21:23:46 - ted: At the moment I'd prefer to store only 7 bit ASCII and use HTML encoding for everything else
21:24:41 - ted: This should work regardless of any codepage setting of the database or OS
21:24:43 - GuKKDevel: I'm not so deep in that things, with configurations and so
21:25:43 - ted: I also don't know exactly how the database and PHP handle non-ASCII
21:26:25 - GuKKDevel: ahh I found in the bug that wytze configured iso-8859-1
21:28:20 - ted: Yes, it looks like, at least on the testserver database, "ö" is stored in iso-8859-1.
21:29:03 - ted: But, this has nothing to do with the new certificates, that's part of the migration job, correct?
21:29:19 - GuKKDevel: I think so
21:30:22 - ted: OK, so, firstly you should give me a CAP form with SHA1 fingerprints.
21:30:53 - GuKKDevel: will do it in the next days
21:30:54 - bdmc [bdmc@user-0c8huj1.cable.mindspring.com] hat den Raum betreten.
21:31:20 - bdmc: Sorry to be late.
21:31:33 - GuKKDevel: hi brian
21:31:54 - bdmc: 'Morning, Karl-Heinz
21:31:54 - ted: Then we should try to get non-ASCII data stored in the database as HTML encoded. I guess that's the best way to migrate to the final goal of UTF-8
21:31:59 - ted: Hi Brian!
21:32:25 - bdmc: 'Evening, Ted.
21:33:00 - ted: I'm afraid that we'll have to run some database scripts on the production DB to fix those encoding issues... :-(
21:33:24 - GuKKDevel: I'm not eager to store non-ASCII.
21:34:05 - bdmc: Another option, and it may be no better, is to migrate to a "new" database, with the correct encoding.
21:34:12 - GuKKDevel: I just thought about the error as in bug-1441
21:34:23 - ted: Hmm, let me try something on the testserver...
21:35:23 - GuKKDevel: for fixing these encoding issues we are waiting for an arbitrator to give us allowance
21:36:04 - ted: Just want to try how something like Walesa is stored. That's non-Latin-1
21:38:00 - ted: $&%/
21:38:16 - ted: Mail address verification prohibits registration of new accounts...
21:38:50 - bdmc: Does the mail system allow non-ASCII?
21:38:57 - bdmc: For addresses?
21:39:24 - ted: The problems are not the mail addresses
21:39:30 - bdmc: OK
21:39:35 - ted: those should all be ASCII#
21:39:46 - bdmc: Names and things like that, correct?
21:40:30 - ted: Yes. Most of all names, and to a smaller amount comments, for example on certificates.
21:41:15 - bdmc: Do we allow that in our advertised rules?  ( non-ASCII? )
21:41:47 - ted: At least on the testserver I'm registered as "Fröhlich"
21:42:39 - bdmc: Is that explicit in the rules, or just because it is "not forbidden?"   It's been too long since I studied those rules.
21:43:26 - bdmc: PHP's "natural language" is utf-8.
21:43:36 - ted: The rules encourage using of "correct" characters, but allow common substitutions like ö -> oe (in german)
21:44:33 - bdmc: So, as long as we can Input and Output a character, we should also store it. ( must )
21:45:15 - bdmc: How is the Arbitration situation coming?
21:46:18 - GuKKDevel: bdmc: are you in contact with the treasurer?
21:46:29 - ted: So, I just checked, at least in comments non Latin-1-characters are stored in the database as HTML encoded. Which is OK.
21:46:48 - bdmc: No.  I saw your message, and wondered whether you had got a response.  Have you tried Etienne?
21:47:14 - GuKKDevel: bdmc: no not yet
21:47:52 - bdmc: GuKKDevel: I sent him a message about a week ago, you might have seen it, and have had no response either.
21:48:16 - ted: So "Walesa" is stored as "Wałęsa". But when displaying the data the ampersand is itself encoded to & and so the comment is displayed garbled, exactly as it is in the database.
21:48:49 - bdmc: ted: OK, so does this get us --- so input and output are not commutative?
21:49:15 - ted: Looks like.
21:49:39 - bdmc: That sounds like an error in htmlentities() or one of the other translation functions.  ( mis-use of that pair of functions )
21:49:49 - ted: But IMHO this is no big issue, because it can be fixed by simple code changes, and the data in the database is correct.
21:50:13 - bdmc: As long as it is going IN to the database correctly.
21:50:23 - ted: Exactly.
21:50:51 - bdmc: You should be able to use the MySQL command line to confirm that.
21:51:15 - bdmc: Which you already did.
21:51:23 - ted: Probably the other characters which are part of Latin-1, as ÄÖÜ, will be more trouble when trying to migrate to UTF-8 consistently.
21:52:00 - bdmc: Do they not have appropriate utf-8 equivalents?
21:53:14 - GuKKDevel: we have a function utf8_to_ascii
21:53:32 - ted: That's not the problem, but they are stored in the database as one single byte.
21:53:52 - bdmc: Sounds like we might need "latin1_to_utf8()".
21:54:02 - GuKKDevel: which creates a database for transforming UTF-8 into ASCII
21:54:05 - ted: So those have to be changed in the database bevore we can switch the database "codepage" to UTF-8
21:54:24 - ted: Excactly. :-)
21:54:47 - bdmc: I would rather copy to new tables, instead of trying to "convert in place."  ( new tables in a new database )
21:54:52 - ted: Or "latin1_to_HTML()"
21:55:19 - bdmc: That way we don't damage the original database.  ( don't have a chance to do so )
21:55:48 - ted: Yes, probably this will be best, at least for important tables like USERS.
21:56:05 - ted: I guess we can live with a few garbled comments.
21:57:02 - bdmc: I haven't tried to do "insert into X ( select * from Y )" where X and Y are in different databases with different encodings.  ( different databases, just not different encoding )
21:57:32 - ted: That's the reason why I'd prefer HTML encoding as an intermediate.
21:58:14 - bdmc: But if my statement works, then we let MySQL do all of the heavy lifting, and don't try to code around it.
21:58:35 - ted: You can store HTML encoded data (which is 7 bit ASCII) in the table, then change the database encoding to UTF-8
21:59:21 - bdmc: Understood, and yes, that is a good idea, if you don't want "native encoding."   The HTML will always be HTML, no matter what the database encoding is.
21:59:35 - ted: Yes.
21:59:53 - ted: And if we want to get it as native-UTF-8 it's easy to do.
22:00:04 - ted: Later...
22:00:30 - bdmc: If we want to convert the Latin1 characters the UTF-8 characters --- OK.  If you want to delay things.
22:00:52 - bdmc: ( the UTF ==> to UTF )
22:01:35 - ted: I strongly assume that we already have HTML encoded characters in the database, since al those eastern Europe people are not covered with Latin-1
22:02:21 - ted: Czech, Polish, Turkish, ...
22:02:53 - bdmc: True.  So the system was already doing HTML encoding, "naturally?"
22:03:19 - ted: Yes, looks like.
22:03:36 - ted: What cannot be encoded in Latin-1 gets stored as HTML
22:04:06 - bdmc: OK.  Hmmm.  How would it tell?  Test the PHP character for a certain range of values?
22:04:37 - bdmc: I don't think that MySQL would do this on its own.
22:04:51 - ted: I'm not sure at which level this is handled. May be already in the browser, or may be PHP.
22:05:10 - ted: MySQL is most probably not doing anything.
22:05:11 - bdmc: Interesting.  More research.
22:05:23 - GuKKDevel: I noticed the use of htmlentities
22:06:09 - GuKKDevel:  htmlentities()
22:06:31 - bdmc: Yes, that is the function, and html_entity_decode(), if it is done correctly.
22:06:37 - ted: I know that, if you set Latin-1 as charset in the HTML header, already the browser sends Non-Latin-1 as HTML entities.
22:06:55 - ted: But it looks like (at least the testserver) sets utf-8 in the HTML header
22:07:08 - bdmc: ted: Ahhh.  Thank you.  So it is happening outside our code.
22:08:17 - bdmc: But that means that if we try to encode things, in the same way, we might ( probably ) get conflicts, like that mis-transalated "&".
22:09:25 - bdmc: As usual, we need to CAREFULLY handle Input and Output.
22:10:11 - ted: The production server seems to use "windows-1252" in the HTTP-header, the HTML header does not specify a character set. So indeed I guess that in the productive system the browser already does the work.
22:11:30 - bdmc: Good old "windows-1252."
22:11:50 - bdmc: Not exactly Linux.
22:12:18 - ted: As usual in the area of character encoding, things are quite messy... :-\
22:14:31 - ted: Now for something completely different...
22:14:57 - GuKKDevel:  another question; I trying to document the current system. Now I have a problem with understanding the flow logic.
22:15:09 - ted: I guess that it would make sense to disable the mail address checking when registering on the testserver.
22:15:34 - ted: OK,. GuKK, what's your problem?
22:15:42 - GuKKDevel: if I enter www.cacert.org I will get to index.php
22:16:05 - GuKKDevel: in index php there is an include to two other files
22:16:52 - bdmc: general.php, in particular.   It comes in from the .htaccess file, as well.
22:17:10 - GuKKDevel: then there is a function call loadem, tht function is in includes/general.php but I can't find the place where this file is included
22:17:37 - GuKKDevel: ahhhh thanks bdmc
22:17:39 - ted: Yes. Me neither.
22:17:40 - bdmc: includes/general.php, which calls includes/lib/general.php,
22:18:15 - ted: .htaccess is the culprit???
22:18:19 - bdmc: that .htaccess got me too.   I don't like it.  I would like to find a different way to do that.
22:19:12 - ted: Which .htaccess?
22:19:13 - bdmc: It probably wouldn't be difficult.  We would just need to agree.
22:19:27 - bdmc: The one in "root".
22:19:34 - bdmc: ( above www )
22:20:14 - /home/cacert ist kein unterstützter Befehl. Geben Sie /help ein, um eine Liste verfügbarer Befehle zu sehen.
22:20:19 - ted:  /home/cacert or /home/cacert/www (there is none in both on the testserver)
22:20:32 - bdmc: Sorry.  It is IN www.
22:20:58 - bdmc: If you don't have ".htaccess" in www, your installation is incomplete.
22:21:23 - ted: Yes, you're right. One bonus point for you! :-)
22:21:48 - GuKKDevel: is in /home/cacert/www/www
22:21:50 - bdmc: Awww.  You're too kind.
22:22:09 - bdmc: I don't know how your directories are laid out.
22:22:21 - ted: OK I hate this trick. Let's get rid of it in the course of the migration to PHP 7
22:22:40 - GuKKDevel: the first www is the mirror link to the git
22:23:51 - bdmc: There are two different things, like this, in the .htaccess file.  There is also something that I would expect to find in the Apache configuration, pointing at the CPS.
22:23:58 - ted: The like "php_value auto_prepend_file /www/includes/general.php" looks very much like the culprit.
22:24:13 - bdmc: That's the one.
22:24:34 - ted: The redirect is also not really nice.
22:24:41 - bdmc: Agreed.
22:25:10 - bdmc: That should ( at least in my opinion ) be in the Virtual Host configuration.
22:25:37 - bdmc: ( or general configuration )
22:25:44 - ted: Or do it in cps.php irself.
22:26:06 - ted: Why hide it in configuration when it's content?
22:26:08 - bdmc: What this does is "remove" cps.php.
22:26:20 - ted: Yes, of course.
22:26:57 - GuKKDevel: this is a way to undergo the necessity to have changes checked
22:27:01 - bdmc: But, I agree.  Why "rename" the HTML file with this redirect?  Just call the HTML file cps.php, and be done with it.
22:27:54 - bdmc: GuKKDevel: I don't understand.  Don't all files need to be checked at some point?
22:27:54 - GuKKDevel: the file cps.php is under control of software and critical, the other file is not
22:28:16 - bdmc: Ahhh.  OK.  Hmmm.
22:28:24 - ted: There may be good reasons why the HTML file is not renamed. I'm afraid that GuKK is right.
22:28:42 - bdmc: So the files in "policy" are not "critical?"
22:28:56 - GuKKDevel: there can policy-group do the changes and doesn't have to wait for a software
22:29:26 - bdmc: OK.  So create cps.php and have it "include" the "policy" file.
22:29:53 - bdmc: That way, any changes in Policy are immediately refected in cps.php, but no code changes.
22:30:04 - GuKKDevel: but in that case policy is still out of control of software
22:30:29 - bdmc: I thought that that was the goal.
22:30:48 - GuKKDevel: I don't know what the goal was
22:31:14 - ted: Maybe the reason is just that CertificationPracticeStatement.html is edited in the version management under this name. And it IS a better name than cps.php.
22:32:07 - bdmc: Which is reasonable.  is "cps.php" called internally?
22:32:50 - ted: I guess (don't know for sure) that cps.php is considerably oder.
22:32:58 - ted: oder -> older
22:33:41 - ted: CertificationPracticeStatement.html was edited by policy group, only "a few years ago"
22:34:15 - bdmc: Hmmm.  I just searched the code, and I don't find a reference to cps.php.
22:34:58 - ted: It may be in the WiKi or on external websites
22:35:24 - ted: No way to be sure that there are none remaining!
22:35:40 - ted: (others than looking for 404 errors)
22:36:01 - bdmc: I went up a level, and found a couple of references.  More to investigate.  ( later )
22:37:24 - GuKKDevel: cps.php is called in pages/account/3.php
22:37:42 - ted: So, let's add a dummy cps.php which just includes the HTML file, and remove the .htaccess magic.
22:38:05 - ted: No big deal
22:38:08 - bdmc: pages/account/3.php, if anybody is interested.
22:38:36 - bdmc: GuKKDevel: you beat me.  B-)
22:38:46 * GuKKDevel grins
22:40:57 - GuKKDevel: but includes/general.php should stay, because we had to include it in all other files prophylactical
22:41:22 * ted sighs deeply.
22:41:57 - bdmc: True, but we can do that in a different way than .htaccess.
22:43:43 - bdmc: We only need to do this in the "start" programs.  The programs that are included in those "start" programs don't need general.php.
22:44:26 - ted: Yes, so probably only the files in www/www, without the subdirectories.
22:45:03 - ted: And maybe api and cats (though I doubt that the cats api needs general.php)
22:45:17 - bdmc: Should be fewer than that.  Perhaps only index.php, but I would want to check that.
22:46:08 - bdmc: ted: You're probably right.
22:46:32 - GuKKDevel: cats needs sanitize from general.php
22:49:24 - GuKKDevel: wrong has its own function defined
22:49:33 - ted: GuKK: It only needs sanitize_string, which is implemented in cats_import.php
22:50:05 - ted: But anyway, I'd really prefer to explicitly include that file.
22:50:17 - ted: (at least when it is needed)
22:50:25 - GuKKDevel: so do I
22:50:45 - bdmc: Agreed
22:51:14 - ted: It does quite many things. I now dimly remember that I also found this out some time ago...
22:51:14 - GuKKDevel: and as I'm documenting, I could do the include to each file that needs a function from general
22:51:39 - ted: And since I forgot it agains it is important to be documented! :-)
22:52:04 - bdmc: Documentation -- who needs documentation?   B-)
22:52:27 * ted giggles inanely.
22:52:45 - bdmc: Combination of general and lib/general.
22:52:53 - ted: Can it be that there are problems when general.php id included twice?
22:52:57 - bdmc: And the other files that they include.
22:53:19 - bdmc: require_once() would take care of that.
22:53:55 - ted: Yes, but it would explain the problems I had when playing around with your bug branch, Brian.
22:55:28 - bdmc: Hmm.  Good point.  Yes, I had included it manually, during my own testing, and it somehow got included in the committed files.
22:55:35 - ted: Nope, there was a "require_once"
22:56:29 - bdmc: require_once() makes sure that an include file is only included once.
22:57:12 - GuKKDevel: see I correct, include also as require_once includes the file at the place where the include/require-statement is?
22:57:17 - ted: Whatever... maybe "auto_prepend_file" is not counted as an include...
22:57:51 - bdmc: It would seem to me that that would do it whether you wanted it to or not.
22:57:58 - GuKKDevel: and if there is a logic in that file it is executed?
22:58:28 - ted: Like it is in general.php
22:59:18 - bdmc: GuKKDevel: Yes, includes and requires happen at the point in the source file where they occur.  Also yes, if there is executable code, not just functions, that code is executed at that point.
22:59:37 - GuKKDevel: thanks
23:00:28 - ted: So, this is the reason why it is needed in every top-level file. There's some setup code in there.
23:01:07 - bdmc: True.  But including it at the top of index.php would take care of this issue.
23:02:04 - GuKKDevel: so as there are only functions defined in includes/general.php and includes/lib/general.php there should be no problem to change the includes to require-once
23:02:30 - bdmc: yes
23:03:02 - GuKKDevel: and add a require_once for safety if a module from one of these files is used
23:03:06 - ted: Include/general.php indeed does execute code!
23:03:53 - ted: The first 100 lines of it are ouside of any function definition
23:04:06 - GuKKDevel: I see
23:04:42 - ted: It includes a whole lot of things...
23:04:57 - GuKKDevel: so should we leave the auto-prepend statement and document it only in codedocs?
23:05:00 - ted: and set session variables
23:05:42 - ted: I really don't know what to do...
23:06:41 - GuKKDevel: all things done there would go to the new class brian developes
23:06:50 - bdmc: ted: That's all right.  As long as that "setup" code is only executed once, at the beginning of execution, then it is done correctly.  ( UNLESS that code needs to re-execute for some bizzarre reason. )
23:07:15 - GuKKDevel: so why not wait until we can install this class?
23:07:32 - ted: Maybe that's the trick...
23:07:47 - bdmc: At the moment that class is for Configuration, but I suppose that we can do setup there, too.
23:08:20 - GuKKDevel: or may be another class only for setup
23:08:48 - bdmc: Sorry.  Yes, that was what I should have said.
23:09:45 - ted: So I think, postponed for now?
23:09:53 - GuKKDevel: postponed
23:09:54 - ted: Just do the docu
23:10:03 - bdmc: OK
23:10:09 - GuKKDevel: OK
23:10:56 - ted: ANyone still wants to talk about testserver-setup? I'm getting a bit tired..
23:11:00 - GuKKDevel: I will do he docu for .htaccess as soon as possible
23:11:28 - bdmc: ted: I'm good for about six hours.   B-)
23:11:47 - bdmc: But I am quite willing to close.
23:11:56 - GuKKDevel: so am I
23:12:23 - ted: I guess it won't make sense to schedule another meeting this year.
23:12:53 - bdmc: Up to you two, and others.  I'm not going anywhere.
23:13:38 - GuKKDevel: lets try 2018-12-21
23:13:55 - ted: How about next meeting at Jan 04? Or better Jan 11? Just in case we can have informal meetings inbetween. :-)
23:14:02 - bdmc: Fine with me, either way.
23:14:12 - ted: OK, then we'll tra Dec 21.
23:14:35 * ted thinks carefully.
23:15:14 - GuKKDevel: OK and  schedule the next for 04.1.19 in advance
23:15:23 - ted: I won't be there Dec 21. But that should not disturb you.
23:15:44 - GuKKDevel: if we do not meet, merry chrismas and a happy new year
23:16:15 - bdmc: I can put them both in my calendar.
23:16:31 - GuKKDevel: but we communicate on devel list
23:16:38 - bdmc: We can definitely do that.
23:16:41 - ted: And have a nice time sweating at the christmas tree, down under!
23:17:00 - ted: I'll try to do some minutes. See you next year!
23:17:33 - GuKKDevel: so good bye for now
23:17:37 - bdmc: Have a very Happy Christmas, all!
23:17:58 - ted: And to all of you!
23:18:12 - GuKKDevel: same from me