Sphinx 1.8.4 - Build Process Error

[mimeiners]

Dear Doconce Users,

For some time the command 'doconce sphinx_dir theme=$theme $name' causes errors.
I need to run it twice to obtain automate_sphinx.py in my work dir.
It seems that sphinx-build has been changed and path variables behave different.
Has someone experienced similar behaviour?
Shall I track this inside the relevant doconce class?

Regards

Mirco Meiners

[KGHustad]

I have also noticed that sphinx hasn't been working as it should lately. DocOnce's integration with Sphinx is quite fragile, and can easily break when changes are made to Sphinx. I don't have the time to look into it now, but if you want to attempt to fix it, then that's great. Just remember to do the fix in a separate branch and create a pull request so that I can review it before it's merged.

[mimeiners]

Alright, I will have a look into the class and the sphinx-build process.
I will open a sphinx branch and keep you informed about my fixing.

[mimeiners]

Dear Kristian,

I had a look into the Sphinx process and I was able to identify the relevant code segments.
As you mentioned, Sphinx implementation in Doconce is quite fragile and any modification to sphinx-quickstart by its developers causes a fail in Doconce, since sphinx_dir is a system-call to sphinx-quickstart.

I tested sphinx_dir calls under Python 2.7 with Sphinx 1.8.5 and Python 3.7 with Sphinx 2.1.2.
The questionary in sphinx-quickstart version 2.1.2 is much shorter than in previous Sphinx versions, hence tmp_sphinx_gen.sh has to be modified accordingly. All of this is done in bin/doconce from line 364 on.

How should we proceed, since this is a question of Python 2.7 to Python 3.x, too? Right?
For Python 2.7 it is a minor fix, for Python 3.x it includes porting of automake_sphinx.py, too.

Summer greetings

Mirco

[KGHustad]

I took a quick look at the sphinx-quickstart source code:
https://github.com/sphinx-doc/sphinx/blob/master/sphinx/cmd/quickstart.py

If we could provide all information via command line arguments to sphinx_dir, then that is probably a better solution than hardcoding answers to the questions asked by the prompt as we have seen that these can change between versions. Maybe we can't provide all the information via command line arguments, but we should least use them where possible, and then there might be a few things that have to be answered via the prompt.

Another approach would be to call directly into the Sphinx API. It seems like the function handling the interactive prompt simply adds keys to a dictionary, so we could probably populate that dictionary in DocOnce, and then call the generate function in the file linked above.

I won't have time to fix this myself anytime soon, but if you're willing, I think you could find a solution with one of these two approaches.
As for Python 2/3 differences, I'm not too concerned about fixing this for Python 2, as Python 2 is nearing EOL.

[mimeiners]

Hallo Kristian, Thanks for your quick comments. Okay, I will give it a try to fix the Sphinx issue. I will review both solution paths and I'll keep you informed about my ongoing. As already agreed, I will use branch sphinx for testing. Best regards Mirco

…

Am 01.08.2019 um 18:58 schrieb Kristian Gregorius Hustad ***@***.***>: I took a quick look at the sphinx-quickstart source code: https://github.com/sphinx-doc/sphinx/blob/master/sphinx/cmd/quickstart.py <https://github.com/sphinx-doc/sphinx/blob/master/sphinx/cmd/quickstart.py> If we could provide all information via command line arguments to sphinx_dir, then that is probably a better solution that hardcoding answers to the questions asked by the prompt as we have seen that these can change between versions. Maybe we can't provide all the information via command line arguments, but we should least use them where possible, and then there might be a few things that have to be answered via the prompt. Another approach would be to call directly into the Sphinx API. It seems like the function handling the interactive prompt simply adds keys to a dictionary, so we could probably populate that dictionary in DocOnce, and then call the generate function in the file linked above. I won't have time to fix this myself anytime soon, but if you're willing, I think you one of these two approaches. As for Python 2/3 differences, I'm not too concerned about fixing this for Python 2, as Python 2 is nearing EOL. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#176?email_source=notifications&email_token=ABSMECIHNELQSXK4TQB5G7TQCMIZ7A5CNFSM4GYKILN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3LHUEY#issuecomment-517372435>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABSMECP4OVH7XNF3P5S5LSDQCMIZ7ANCNFSM4GYKILNQ>.

[mimeiners]

Dear Kristian,

currently I prepare my lecture slides for the coming winter term. I use this scenario to work on the sphinx issue in doconce.

As my lecture slides are in German as well as in English I got stucked due to
unicode issues. This becomes unstructured, if I switch between python 2.7 and
python 3.7; esp. module re (regex) in python 3.7 causes error messages during doconce calls format latex.

I am writing this to you to sync appropriately into your doconce workflow.

How can I best support your work and push modification into the repositorium?
What is your approach to switch the code from python 2.7 to python 3.x?
Working on Sphinx format results in rewriting some functions of doconce, e.g.
automake_sphinx.py.

Best regards

Mirco

[KGHustad]

I'm not sure that I understand your questions exactly, but I will try my best to answer.

As my lecture slides are in German as well as in English I got stucked due to
unicode issues. This becomes unstructured, if I switch between python 2.7 and
python 3.7; esp. module re (regex) in python 3.7 causes error messages during doconce calls format latex.

Could you post the error message and traceback?

How can I best support your work and push modification into the repositorium?

Create a separate git branch, commit the changes there, and push that branch either to this repo (please don't push directly to master) or to your own fork, and then create a pull request

What is your approach to switch the code from python 2.7 to python 3.x?

We have used autotranslation tools previously, but they don't fix the regular expressions, which is the key issue in our case. My approach for the regular expressions have been to fix them by hand. Generally what needs to be done is to double-escape \ characters. '\\' is a string with a single backslash. '\\\\' is a string with a double backslash, which re will interpret as a single literal backslash.

[mimeiners]

I'm not sure that I understand your questions exactly, but I will try my best to answer.

You made a great job and answered quite well ;o)

As my lecture slides are in German as well as in English I got stucked due to
unicode issues. This becomes unstructured, if I switch between python 2.7 and
python 3.7; esp. module re (regex) in python 3.7 causes error messages during doconce calls format latex.

Could you post the error message and traceback?

I will do that.

How can I best support your work and push modification into the repositorium?

Create a separate git branch, commit the changes there, and push that branch either to this repo (please don't push directly to master) or to your own fork, and then create a pull request

Alright, made two braches, sphinx_v1 and sphinx_v2. sphinx_v1 is the branch to which I committed my latest modification dealing with utf-8 (unicode) for German language with python 2.7 and sphinx vers. 1.8.5.

Branch sphinx_v2 holds modification for python 3.7 and sphinx vers. 2.1.2

What is your approach to switch the code from python 2.7 to python 3.x?

We have used autotranslation tools previously, but they don't fix the regular expressions, which is the key issue in our case. My approach for the regular expressions have been to fix them by hand. Generally what needs to be done is to double-escape \ characters. '\\' is a string with a single backslash. '\\\\' is a string with a double backslash, which re will interpret as a single literal backslash.

Yep, this is what I meant when writing about errors during doconce call for format latex.
The behaviour of module re is different in py 2.7 and py 3.7 which causes errors processing language German in doconce, I called it unicode errors.

Okay, I will post the error, but in the meantime you might have a look into my branches.

[KGHustad]

I had a look at your branches.

I saw you made some improvements to German language support, so I made a separate commit with these:
a68daba

There is no need to specify the encoding of the .py files for Python 3. Also, I would rather not have """ Module description. """ strings at the top of the files.

Does sphinx work for you with the changes you have made in these branches?

[mimeiners]

__

I had a look at your branches.

I saw you made some improvements to German language support, so I made a separate commit with these:
a68daba

Yep!

There is no need to specify the encoding of the .py files for Python 3. Also, I would rather not have """ Module description. """ strings at the top of the files.

Okay!

Does sphinx work for you with the changes you have made in these branches?
Yep, with the modifications in branch sphinx_v1 it works with python 2.7 and sphinx.

If I run doconce with python 3.7, sphinx runs with a work-around in my make.sh:

system python2 automake_sphinx.py

In my next step, I will translate that file to python 3.
Maybe it is no longer necessary, if I am able to link with a dictionary into sphinx-build (one of your ideas to optimise sphinx processing).

But doconce format pdflatex gives the following error with python 3.7:

Traceback (most recent call last):
File "/opt/local/bin/doconce", line 1690, in
bg_session = main()
File "/opt/local/bin/doconce", line 1679, in main
retval = eval(command + '()')
File "", line 1, in
File "/opt/local/bin/doconce", line 103, in format
bg_session = doconce.doconce.format_driver()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/doconce.py", line 5543, in format_driver
out_filename, bg_session = file2file(filename_preprocessed, format, basename)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/doconce.py", line 4355, in file2file
filestr, bg_session = doconce2format(filestr, format)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/doconce.py", line 4847, in doconce2format
tex_blocks, format)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/latex.py", line 643, in latex_code
pattern = re.compile(r'!b%s *\n', re.UNICODE) % admon
TypeError: unsupported operand type(s) for %: 're.Pattern' and 'str'

• '[' 1 -ne 0 ']'
• echo 'make.sh: unsuccessful command doconce' format pdflatex lec --latex_title_layout=beamer --latex_code_style=pyg --latex_movie=href --encoding=utf-8 --siunits --language=German
  make.sh: unsuccessful command doconce format pdflatex lec --latex_title_layout=beamer --latex_code_style=pyg --latex_movie=href --encoding=utf-8 --siunits --language=German
• echo 'abort!'
  abort!
• exit 1

[KGHustad]

This is the line that is failing

doconce/lib/doconce/latex.py

Line 642 in a7fef2b

pattern = r'!b%s *\n' % admon

From your traceback, I see that this line is set to

pattern = re.compile(r'!b%s *\n', re.UNICODE) % admon

in your local latex.py. I assume this is a change you made in attempt to fix the problem, but I cannot provide much help in debugging code that I cannot see.
Could you check if converting this file to latex works with the latest version of DocOnce?

[mimeiners]

latest DocOnce version

• theme=hsb_blue
• system doconce format pdflatex lec --latex_title_layout=beamer --latex_code_style=pyg --latex_movie=href --encoding=utf-8 --siunits --language=German
• doconce format pdflatex lec --latex_title_layout=beamer --latex_code_style=pyg --latex_movie=href --encoding=utf-8 --siunits --language=German
  running preprocess -DFORMAT=pdflatex -DDEVICE=screen lec.do.txt > tmp_preprocess__lec.do.txt
  translating doconce text in tmp_preprocess__lec.do.txt to pdflatex
  open file with encoding utf-8
  *** locale set to German
  Final all \label{(.+?)}

exporting publish database papers.pub to papers.bib:
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/sre_parse.py", line 1021, in parse_template
this = chr(ESCAPES[this][1])
KeyError: '\e'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/local/bin/doconce", line 1690, in
bg_session = main()
File "/opt/local/bin/doconce", line 1679, in main
retval = eval(command + '()')
File "", line 1, in
File "/opt/local/bin/doconce", line 103, in format
bg_session = doconce.doconce.format_driver()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/doconce.py", line 5543, in format_driver
out_filename, bg_session = file2file(filename_preprocessed, format, basename)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/doconce.py", line 4355, in file2file
filestr, bg_session = doconce2format(filestr, format)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/doconce.py", line 4847, in doconce2format
tex_blocks, format)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/doconce/latex.py", line 664, in latex_code
filestr = re.sub(pattern, r'\end{block}', filestr)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/re.py", line 192, in sub
return _compile(pattern, flags).sub(repl, string, count)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/re.py", line 309, in _subx
template = _compile_repl(template, pattern)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/re.py", line 300, in _compile_repl
return sre_parse.parse_template(repl, pattern)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/sre_parse.py", line 1024, in parse_template
raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \e at position 0

• '[' 1 -ne 0 ']'
• echo 'make.sh: unsuccessful command doconce' format pdflatex lec --latex_title_layout=beamer --latex_code_style=pyg --latex_movie=href --encoding=utf-8 --siunits --language=German
  make.sh: unsuccessful command doconce format pdflatex lec --latex_title_layout=beamer --latex_code_style=pyg --latex_movie=href --encoding=utf-8 --siunits --language=German
• echo 'abort!'
  abort!
• exit 1

[mimeiners]

I had a look at your branches.

I saw you made some improvements to German language support, so I made a separate commit with these:
a68daba

There is no need to specify the encoding of the .py files for Python 3. Also, I would rather not have """ Module description. """ strings at the top of the files.

Does sphinx work for you with the changes you have made in these branches?

master branch does not work neither sphinx (py27) version 1.8.5 nor sphinx (py3) version 2.2.0

I committed and pushed necessary modifications into branches sphinx_v1 and sphinx_v2.
It is just a quick and dirty work-around inside bin/doconce for different sphinx-quickbuild calls in version 1 and version 2, as I made for previous sphinx version with Hans-Petter.
I am working on a more robust implementation as we discussed.

[KGHustad]

I think this should fix the error you posted with latex.
9c8fc12