minimapLabelsTemplates.xc has broken character encoding

[renhanxue]

In my install of the modpack's XVM config (logs attached), minimapLabelsTemplates.xc is supposed to be UTF-8, but it contains broken byte sequences (see attached screenshot) which break Python's utf8 decoder, resulting in the following error in xvm.log during the config loading (the end result of which is a lot of XVM functionality going missing in game):

2015-12-21 21:10:40: [ERROR] Traceback (most recent call last):
  File "xvm_main/config.py", line 97, in _load_xvm_xc
  File "mods/xfw/python/lib/JSONxLoader/__init__.py", line 9, in load
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 25, in load
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 44, in visit
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 69, in visit_dict
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 44, in visit
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 69, in visit_dict
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 44, in visit
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 52, in visit_dict
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 52, in <dictcomp>
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 44, in visit
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 69, in visit_dict
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 44, in visit
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 52, in visit_dict
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 52, in <dictcomp>
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 44, in visit
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 69, in visit_dict
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 44, in visit
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 52, in visit_dict
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 52, in <dictcomp>
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 46, in visit
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 78, in visit_list
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 44, in visit
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 58, in visit_dict
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 33, in load_config
  File "mods/xfw/python/lib/JSONxLoader/loader.py", line 88, in load_file
  File "scripts/common/Lib/codecs.py", line 668, in read
  File "scripts/common/Lib/codecs.py", line 474, in read
  File "scripts/common/Lib/encodings/utf_8_sig.py", line 104, in decode
UnicodeDecodeError: 'utf8' codec can't decode byte 0xb8 in position 41: invalid start byte

Since the broken byte sequences are in comment lines only it's perfectly safe to just remove the offending lines if you don't want to fix the Russian.

 

Someone should go tell the XVM guys that they should probably catch exceptions from the config file loading and print out the offending filename in addition to the original stack trace in the log - I had to write a Python script to find out minimapLabelsTemplates.xc was the culprit, since the stack trace currently printed to the log doesn't actually tell you which file it failed to load.

 

In case you're ever in this situation again, here you go:

import os

os.chdir(r"C:\World_of_Tanks\res_mods\configs\xvm\Aslain")
for fn in os.listdir("."):
	if not fn.endswith(".xc"):
		continue
	with open(fn, "r") as f:
		print "about to load {}".format(fn)
		foo = f.read().decode('utf-8') # throws an exception if the file wasn't proper utf8
		

_Aslain_logs.zip

Edited December 21, 2015 by renhanxue

[thevbm]

In past sometimes .xc were saved using BOM which lead to broken files. Have you tried re-saving your .xc without BOM marks ?

[renhanxue]

Some have a BOM, some don't, both work these days. I don't see how it's relevant though since I already diagnosed and fixed the problem. I'm just reporting it so it can get fixed in the modpack and won't trip anyone else over. I haven't made any edits to the .xc's myself other than removing these broken lines and everything works again now, but if it doesn't get fixed in the modpack it'll just re-break when I update.

Edited December 21, 2015 by renhanxue

[Aslain]

Forgot to convert to UTF-8 w/o BOM in few new files, and when diff-patcher comes, it's breaking the file. Will fix in next modpack.