[Docker] Initialization fails for nld-vnn and dum

[tcbrouwer]

I run Frog from https://hub.docker.com/r/proycon/frog:latest . Currently running

docker run proycon/frog --language=dum

results in

frog-:config read from: /usr/local/share/frog/dum/frog.cfg
frog-:Missing [[mbma]] section in config file.
frog-:Disabled the Morhological analyzer.
frog-:Missing [[IOB]] section in config file.
frog-:Disabled the IOB Chunker.
frog-:Missing [[NER]] section in config file.
frog-:Disabled the NER.
frog-:Missing [[mwu]] section in config file.
frog-:Disabled the Multi Word Unit.
frog-:Also disabled the parser.
frog-mblem-:Initiating lemmatizer...
ucto: textcat configured from: /usr/local/share/ucto/textcat.cfg
frog-tok-:Language List =[dum]
ucto: No useful settingsfile(s) could be found (initiating from language list: [dum])
frog-tagger-tagger-:reading subsets from /usr/local/share/frog/dum//crmsub.cgn
frog-tagger-tagger-:reading constraints from /usr/local/share/frog/dum//crmconstraints.cgn
frog-:Initialization failed for: [tokenizer] 
frog-:fatal error: Frog init failed

Similar for nld-vnn (after mitigation LanguageMachines/frogdata#7 )

With my previous build from 6 months ago, I did not encounter this initialization problem.

My current mitigation is to use

docker run proycon/frog --language=dum,nld

instead. With this the initialization succeeds, and if I understand correctly from

WARNING: you used the --language=dum,nld option with more then one language 
	 specified. These values will be handled to the tokenizer, but Frog
	 will only handle the first language: dum for further processing!

still only dum is used for Frog, which is the core of our use case.

As a sidenote, in general the dockerized deployment of Frog is very helpful to me.

[kosloot]

Wow. This seems a strange regression.

Ucto assumes the data to be in .../share/ucto/tokconfig-nld-historical
But frog asks to look for .../share/ucto/tokconfig-dum

Running with frog --language=nld-historical is NOT working too, as frog expects a nld-historical.cfg file then

Quick fix: (assuming you have enough rights to do so):
ln -s /usr/local/share/ucto/tokconfig-nld-historical /usr/local/share/ucto/tokconfig-dum

After that, frog will find the tokconfig-dum file.

Needs a more permanent solution.

[kosloot]

Another not-intrusive possibility is to start Frog using the -c or --config option:
frog -c dum/frog.cfg
This will find the correct tokenizer files.

This is probably the only correct way to do what you want.
But your report reveals some more general problem we need to investigate.

[kosloot]

@tcbrouwer To clarify a few things:

the --languages option is used to initialize the ucto tokenizer with 1 or more language configurations.
When you use --languages=dum in Ucto, the tokenizer searches for a setting for dum and will fail to do so.
It cannot run, nor will frog.

When you use --languages=dum,nld in Ucto, the tokenizer wil NOT find a setting for dum but it wil find one for nld.
Ucto, and therefore Frog will happily run with only the 'standard Dutch` tokenizer rules. NOT those for historical dutch.
This is probably not what you wanted or expected. Failing to run would probably more informative.

But there is an oddity in using --language=dum IN Frog as it CORRECTLY selects the .../dum/* configuration files
In that file it should find the line: rulesFile=tokconfig-nld-historical
And then pass nld-historical on to the Tokenizer.
But it fails doing so. That is the real bug. We will address that.

There was also a problem that commandline ucto will clearly warn you when you use an 'unknown' language, but when integrated in Frog, this message was redirected to a debug file. Not directly to the user. This will be fixed too.

[tcbrouwer]

Thanks for the reaction. My statement

"With my previous build from 6 months ago, I did not encounter this initialization problem."

should have been formulated better to not imply regression, my bad for this. I will clarify shortly.

The newer Docker image for proycon/frog did break my previous build in an unsuspected way. Since I suspected an update in my own wrapper was in order, I rewrote it slightly. In my docker-compose.yml, I replaced

entrypoint: frog --skip=mt -c /usr/share/frog/dum/frog.cfg -S 8020

with

# currently the image fails to launch with dum only, so we use nld-vnn,nld as workaround
command: ["--skip=mptnc", "--language=dum,nld", "-S 8020"]

which got the container running again.

Apparently the entrypoint changed, but also the file /usr/share/frog/dum/frog.cfg is no longer valid (unable to read configuration from /usr/share/frog/dum/frog.cfg ). I tried some things to fix it and thought a --language flag would be the appropriate and more idiomatic way to set language/config, but I should have stuck with config files.

The --help is clear about the actual (new?) location of the config files* (default /usr/local/share/frog/nld/frog.cfg) but I got confused by the errors, and didn't look properly at the help.

*Although you seem to suggest using some relative path, which might have avoided this problem in the first place.

I am now able to initialize the containers using the new location of the config files, this should solve the problem for me. If I encounter more problems on integration I will let you know, but this might take a few weeks.

In case you are interested I gave @proycon access to the relevant repo (since he already is an outside collaborators of INT). The relevant files are:

https://github.com/INL/taggers-dockerized/blob/development/Frog/server/Dockerfile
https://github.com/INL/taggers-dockerized/blob/development/docker-compose.yml

[kosloot]

Ok, this clarifies a lot. Using the config file is the better option indeed.
Glad that you are able to proceed.

Nevertheless your request started me thinking and experimenting, and some things can be improved.

• ucto logging information should end up in the frog logging
• ucto supports an 'und' language option to explicitly mark undetectable text, and skip it completely. Frog does barf on it so I will add that possibility.
• I tried to implement a shortcut that interprets --languages=dum as "use the dum configuration" And it works, but the code is a hassle, and probably not worth wile. A clear error message is probably simpler.

I am working on this.

On a sidenote:
The data used to train 'dum' is terrible, if a better version would be available that would be very useful
It is a version of corpus ghysen, I believe

Example from the data:

tie↩nde tiende  N(soort,ev,met-e)
unreadable      ZZZ     SPEC(overig)
&2periods;      &2periods;      LET()
%waredatzake&unreadable;        %waredatzake&unreadable;        LET()

Also this kind of Tagging drives Frog training crazy:

cc'ho   1200-vierwerftwintig    TW(hoofd,zonder,deel-i)
&       1200-vierwerftwintig    TW(hoofd,zonder,deel-i)
vier    1200-vierwerftwintig    TW(hoofd,zonder,deel-i)
waerf   1200-vierwerftwintig    TW(hoofd,zonder,deel-i)
tuintich        1200-vierwerftwintig    TW(hoofd,zonder,deel-f)

The tokenizer for dum splits cc'ho&vierwaerftuintich into cc'ho & vierwaerftuintich
And the frog analyses it as:

1	cc'ho	1291		TW(rang,overig,deel-i)	0.625000							
2	&	1280-12		TW(rang,overig,deel-i)	1.000000							
3	vierwaerftuintich	vierwaerftuintich		N(eigen,zonder)	0.393292

Is this useful?

[tcbrouwer]

Thanks for the response.

your sidenote is currently out-of-scope for me, but I will relay it to my teammates (Jesse de Does, Katrien Depuydt ...), they are definitely interested in improving the quality at any level.

Maybe this issue is not the best place for a communication channel on this topic. What is your preferred way to communicate on this topic? (e-mail would not be preferred for me, maybe we should start a seperate issue, or we can see whether we can give guest access to our Teams instance or something).

[kosloot]

assume this is a non-issue atm